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EXECUTIVE  SUMMARY 

BACKGROUND 

The  Organizational  Effectiveness  Center  and  School  (OEC&S)  at  Fort  Ord, 
California  has  been  the  training  center  for  the  Army's  Organizational  Effective¬ 
ness  Consultants  since  1975.  The  effectiveness  of  such  consultants  appears  to  be 
dependent  upon  "soft  skill"  interpersonal  competencies,  rather  than  specific 
tasks  which  are  performed.  In  1979,  OECicS  and  the  Army  Research  Institute 
contracted  with  McBer  &  Co.  to  develop  a  model  of  the  competencies  which 
distinguish  the  superior  from  the  average  consultant.  Eighteen  of  the  33 
competencies  identified  in  the  McBer  model  were  considered  by  OEC<5cS  to  be 
potentially  trainable  in  the  course.  However,  the  success  of  the  training  requires 
an  understanding  of  the  competencies  and  an  ability,  on  the  part  of  the  trainers, 
to  recognize  if  and  when  the  competencies  are  being  demonstrated. 

In  order  to  provide  a  resource-efficient  means  of  training  the  trainers  to 
recognize  the  performance  of  the  competencies  by  a  student,  McBer  &  Co. 
joined  with  Interactive  Training  Systems,  Inc.,  (ITS)  to  produce  the  Interactive 
Video  Competency  Recognition  Training  System  (IVCRTS).  The  IVCRTS  links 
videotapes  which  portray  consultants  in  action  with  a  microcomputer  in  order  to 
train  the  trainer  to  identify  specific  competencies  and  to  recognize  them  when 
portrayed  in  various  situations. 

Subsequent  to  the  McBer/ITS  initiatives  in  this  area,  OECJcS  asked  the  Training 
Developments  Institute  (TDI)  to  participate  in  and  support  an  effort  to  assess  the 
value  of  the  IVCRTS  in  teaching  competency  recognition. 

After  investigating  the  OEC&S  request  and  coordinating  with  the  Army  Com¬ 
municative  Technology  Office  (ACTO),  TDI  agreed  to  fund  and  monitor  a  formal 
evaluation  of  the  IVCRTS.  In  addition  to  an  interest  in  the  competency 
recognition  issue,  TDI  was  also  interested  in  learning  if  the  IVCRTS  has  potential 
for  application  in  training  other  types  of  skills,  particularly  if  it  could  be  used 
with  a  disk  system  to  allow  for  "real-time"  interaction  capability. 


Due  to  limitations  imposed  by  the  selection  of  videotapes  used  in  this  effort, 
only  15  of  the  IS  consulting  competencies' were  coded  for  recognition  purposes. 

OBJECTIVE 

The  objective  of  this  effort  was  for  McFann,  Gray  &  Associates,  Inc.,  (MGA)  to 
evaluate  the  effectiveness  and  efficiency  of  the  IVCRTS  in  training  trainers  at 
OEC&S  to  recognize  15  of  the  consulting  competencies.  In  addition,  the 
usability  and  acceptability  of  the  inputting  and  coding  system  used  by  course 
developers  and  the  individual  training  management  aspects  of  the  IVCRTS  were 
evaluated. 

IVCRTS  DESCRIPTION 

The  basic  hardware  in  the  system  includes  an  IBM  Personal  Computer,  an  ITS 
2000  Video  Controller,  and  an  Electrohome  Hi-Res  RGB  Video  Monitor.  Inter¬ 
actions  between  the  IBM  computer,  the  ITS  controller  and  the  ITS  software 
enable  the  user  to  interact  with  audio-video  presentations  originating  from  a 
videotape  player. 

The  ITS  software  is  designed  to  facilitate  learning  by  requiring  users  to  connect 
a  set  of  terms  or  concepts  with  visual  images,  i.e.,  by  requiring  a  learner  to 
become  skilled  at  identifying  something  that  is  presented  in  the  visual  mode. 

As  used  in  the  OEC&S  effort,  the  software  programs  were  designed  to  teach 
trainers  to  recognize  when  one  of  the  15  consulting  competencies  was  being 
displayed  by  an  actor  portraying  an  Army  OE  consultant  interacting  with  clients 
in  a  military  setting.  The  15  competencies  were  coded  onto  pre-existing 
videotapes  by  two  members  of  the  OECicS  Training  Develpment  Directorate. 
The  tapes  were  originally  produced  to  explain  the  OE  process  to  TRADOC 
service  school  students.  The  1 5  competencies  are: 


1. 

Functional  Knowledge 

9. 

Aware  of  Self  Impact 

2. 

Self  Confidence 

10. 

Seeks  Multiple  Perspectives 

3. 

Low  Fear  of  Rejection 

11. 

Causal  Thinking 

4. 

Establishes  Professional  Rapport 

12. 

Identifies  Key  Themes 

5. 

Concern  for  Clarity 

13. 

Problem  Focused  Procedures 

6. 

Seeks  User  Input 

14. 

Seizes  Opportunities 

7. 

Uses  Influence  Strategies 

15. 

Results  Orientation 

8. 

Communicates  Clearly 

EVALUATION  ISSUES  AND  DESIGN 


In  general,  this  effort  sought  to  evaluate  the  major  components  of  the  IVCRTS  in 
terms  of:  ease  of  use,  flexibility,  reliability,  effectiveness,  efficiency,  and  user 
acceptance.  The  major  components  of  the  system  are:  the  Inputting  and  Coding 
System,  the  Training  System,  and  the  Training  Management  System. 

The  study  was  conducted  from  December  through  March  1983  at  the  OEC&S  in 
accordance  with  a  test  and  evaluation  plan  developed  by  MG  A  and  approved  by 
OEC&S  and  TDI. 

Twenty  trainers  from  the  Training  Directorate  of  OEC&S  served  as  the  sample 
of  IVCRTS  users.  They  were  expected  to  use  the  IVCRTS  until  they  had  reached 
an  established  training  criterion  of  70%  correct  recognition  of  70%  of  the  15 
competencies. 

Actual  performance  data  were  recorded  by  the  IVCRTS.  Other  data  were 
collected  via  two  survey/questionnaire  instruments  and  structured  interviews. 
The  collection  methods,  sources  and  numbers/types  of  persons  involved  are 
described  below: 


rii 


COLLECTION  METHOD 

SOURCE 

NUMBER 

IVCRTS  Program 

Trainers 

20 

(14  in  Pairs) 

Problem  Survey  Log 

Trainers 

Course  Developers 

20 

4 

IVCRTS  Evaluation  Questionnaire 

Trainers 

Course  Developers 

12* 

4 

Trainers 

12* 

Structured  Interviews 

Managers 

3 

Coders 

2 

*A  subset  of  the  20  trainers  who  used  the  IVCRTS. 

IVCRTS  TRAINING  SESSIONS 

The  basic  options  available  to  the  IVCRTS  user  are:  Introduction  to  the  OE 
Competency  Model;  Exercise  Play  in  either  a  "Name  It"  mode  or  a  "Find  It" 
mode;  Reliability  scores;  Tutor;  and  Rundown.  The  user  interacts  with  the 
system  by  using  a  light  pen  to  touch  appropriate  portions  of  the  video  screen. 

Ideally,  a  user  would  begin  with  the  Introduction  option  to  learn  about  the 
competencies  available  for  display  and  the  behavioral  indicators  for  each. 
Following  this,  a  user  could  select  one  of  three  available  tape  segments 
(Assessment,  Planning,  or  Evaluation)  to  use  in  participating  in  one  of  two 
exercise  options,  "Name  It"  or  "Find  It."  Seven  or  eight  of  the  1 5  competencies 
are  available  for  display  on  each  of  the  three  tape  segments. 

The  "Name  It"  exercise  is  considered  the  easier  because  it  provides  an  audio  cue 
to  indicate  when  a  competency  is  about  to  be  presented  and  the  system  stops  on 


completion  of  each  competency  presentation  to  allow  the  user  an  opportunity  to 
name  the  competency. 

The  "Find  It"  exercise  is  considered  more  difficult  because  there  is  no  audio  cue 
to  signal  presentations,  nor  does  the  system  automatically  stop  at  the  end  of  a 
representation  in  this  mode  to  allow  the  user  to  identify  the  competencies 
presented.  The  user  in  this  exercise  must  stop  the  system  and  make  an 
identification. 

Given  the  difference  between  the  two  exercises,  the  users  were  advised  to  work 
through  the  tapes  using  the  "Name  It"  exercise  mode  before  attempting  the  "Find 
It"  exercise. 

The  Tutor  option  is  available  for  users  who  have  trouble  identifying  the 
competency  presentations.  It  allows  users  to  re-view  the  tape  segments  they  are 
having  trouble  with. 

The  Reliability  Scores  option  provides  users  with  a  cumulative  record  of 
performance,  and  is  the  feature  the  users  refer  to  to  determine  if/when  they 
reach  the  established  criterion. 

The  Rundown  feature  was  designed  to  assist  users  who  consistently  confuse  one 
or  more  competencies  with  others. 

The  users  were  told  to  work  on  the  system  until  they  had  at  least  a  70% 
reliability  score  for  each  exercise  option  available  on  each  of  the  three  tapes. 

Because  complete  anonymity  was  desired,  user  disks  were  coded  to  prevent 
monitoring  of  individual  programs. 


SUMMARY  CONCLUSIONS  AND  RECOMMENDATIONS 


Summary 

The  analysis  of  available  data  led  to  the  following  summary  conclusions: 

1.  The  IVCRTS  is  a  highly  complex  and  sophisticated  training  apparatus 
that  is  potentially  very  useful  in  the  OEC&S  environment,  especially  if 
combined  with  other  training  methods. 

2.  The  IVCRTS  Inputting  and  Coding  System  requires  several  weeks  of 
study  and  hands-on  experience  to  enable  coders  to  use  it  to  its  full 
potential. 

3.  The  IVCRTS  Training  System,  as  it  was  employed  in  this  evaluation 
effort,  did  not  bring  about  the  training  results  desired. 

4.  Factors  which  contributed  most  to  the  training  performance  obtained 
were* 


•  Limitations  inherent  in  the  videotapes  used; 

•  The  nature  of  the  skills  being  trained; 

•  The  subjectivity  involved  in  the  coding  process; 

•  Absence  of  strong  motivation  of  the  sample  of  trainers; 

•  Conditions  under  which  the  evaluation  was  conducted;  and, 

•  Absence  of  needed  data. 


3.  The  IVCRTS  Training  Management  System  does  not  record  data  in 
sufficient  detail  or  appropriate  format  to  adequately  assess  training 


Recommendations 


1.  Continue  Evaluation.  It  is  recommended  that  OECicS  continue  to  evaluate 
the  effectiveness  and  efficiency  of  the  IVCRTS  in  the  OECicS  setting.  The 
majority  of  the  people  who  contributed  to  the  evaluation  effort  appear  to 
think  that  there  is  a  place  for  the  IVCRTS  within  OECdcS.  Most  trainers,  it 
seems,  would  be  interested  in  using  the  IVCRTS  in  combination  with  one  or 
more  of  the  training  methods  now  employed  in  the  classroom.  The  training 
managers  are  particularly  optimistic  about  the  future  use  of  the  IVCRTS, 
particularly  in  the  areas  of  standardization  of  instruction  and  grading.  The 
training  developers  view  the  IVCRTS  as  an  excellent  resource  and  are 
looking  forward  to  expected  improvements  in  the  system. 

2.  Use  Panel  of  Experts  to  Code.  The  apparent  lack  of  success  of  the  IVCRTS 
in  this  case  may  have  resulted  more  from  the  nature  of  the  skills  being 
trained  (i.e.,  the  McBer  competencies)  than  from  the  system  itself.  The 
consultant-client  interaction  is  a  highly  complex  one  that  requires  the 
successful  consultant  to  demonstrate  a  large  number  of  competencies,  often 
simultaneously.  Any  selected  segment  of  sequential  behaviors  is  so  rich 
with  possible  combinations  that  a  forced  choice  of  some  single  and  minute 
part  is  artificial  at  best.  Add  to  this  the  highly  subjective  nature  of  the 
coding  process,  and  the  chances  of  reaching  a  meaningful  level  of  agreement 
diminishes  significantly. 

Nevertheless,  agreement  could  probably  be  improved  upon  in  this  case  if 
more  experienced  people  were  involved  in  the  coding  process.  It  is 
recommended,  therefore,  that  a  larger  panel  of  subject  matter  experts  with 
demonstrated  success  as  both  instructors  and  consultants  be  involved  in  the 
competency  coding  process. 

3.  Relate  Competencies  to  Class  Learning  Objectives.  Learning  objectives  for 
each  block  of  instruction  apparently  do  not  incorporate  specific  com¬ 
petencies  or  competency  based  behaviors.  If,  in  fact,  an  objective  of  the 


Program  of  Instruction  is  to  train  the  essential  competencies  identified  in 
the  McBer  study,  then  it  seems  appropriate  that  learning  objectives  incor¬ 
porate  specific  competencies  as  appropriate  to  each  objective. 

Evaluate  IVCRTS  Through  Student  Use.  Assuming  that  the  competencies 
support  classroom  learning  objectives  and  that  an  expert  panel  can  agree  to 
realistic  coding  procedures,  it  seems  the  IVCRTS  has  more  potential  for  use 
as  a  training  mechanism  for  students  in  the  consultant's  course.  In  addition 
to  assisting  in  the  standardization  of  instruction  and  grading,  the  IVCRTS 
could  be  used  to  compliment  the  existing  methods  of  instruction.  Further¬ 
more,  since  the  students'  activities  are  controlled  and  monitored  closely,  it 
is  likely  that  the  effectiveness  and  efficiency  of  the  IVCRTS  could  be  more 
accurately  assessed  with  such  a  group.  The  students  are  also  more  likely  to 
be  motivated  to  use  the  system  to  its  full  potential. 

Modify  the  Individual  Training  Management  System.  With  more  control  over 
the  users  of  the  IVCRTS  (i.e.,  students  in  the  course)  and  perhaps  with 
additional  program  software,  it  should  be  possible  to  obtain  more  and  better 
data  to  assist  in  evaluating  the  effectiveness  and  efficiency  of  the  system. 
As  a  minimum,  there  needs  to  be  a  record  of  which  competencies  are 
presented  in  each  session,  in  what  order,  and  with  what  frequency.  Addi¬ 
tionally,  the  amount  of  time  devoted  to  each  session  should  also  be 
recorded.  These  data  should  be  available  for  each  individual  and  for  groups 
of  individuals. 

Also,  for  maximum  effect,  students  should  be  required  to  demonstrate 
ability  on  the  system  individually  rather  than  in  pairs. 

Locate/Develop  Other  Videotapes.  Finally,  It  is  recommended  that  addi¬ 
tional  videotapes  be  located  and/or  developed  to  allow  for  both  the  coding 
of  additional  competencies  and  the  coding  of  the  same  competencies  in 
different  settings  or  contexts.  If  possible,  the  videotapes  would  be  of  actual 
consulting  activities  and  would  not  contain  distracting  narrator  commen¬ 
tary. 
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SECTION  b  INTRODUCTION 


BACKGROUND 

The  Organizational  Effectiveness  Center  and  School  (OECicS)  at  Fort  Ord, 
California  has  been  the  training  center  for  the  Army's  Organizational  Effective¬ 
ness  Consultants  since  1975.  The  effectiveness  of  such  consultants  appears  to  be 
dependent  upon  "soft  skill"  interpersonal  competencies,  rather  than  specific 
tasks  which  are  performed.  In  1979,  OEC&S  and  the  Army  Research  Institute 
contracted  with  McBer  &  Co.  to  develop  a  model  of  the  competencies  which 
distinguish  the  superior  from  the  average  consultant.  Eighteen  of  the  33 
competencies  identified  in  the  McBer  model  were  considered  by  OEC&S  to  be 
potentially  trainable  in  the  course.  However,  the  success  of  the  training  requires 
an  understanding  of  the  competencies  and  an  ability,  on  the  part  of  the  trainers, 
to  recognize  if  and  when  the  competencies  are  being  demonstrated. 

In  order  to  provide  a  resource-efficient  means  of  training  the  trainers  to 
recognize  the  performance  of  the  competencies  by  a  student,  McBer  &  Co. 
joined  with  Interactive  Training  Systems,  Inc.,  (ITS)  to  produce  the  Interactive 
Video  Competency  Recognition  Training  System  (IVCRTS).  The  IVCRTS  links 
videotapes  which  portray  consultants  in  action  with  a  microcomputer  in  order  to 
train  the  trainer  to  identify  specific  competencies  and  to  recognize  them  when 
portrayed  in  various  situations. 

Subsequent  to  the  McBer/ITS  initiatives  in  this  area,  OEC&S  asked  the  Training 
Developments  Institute  (TDI)  to  participate  in  and  support  an  effort  to  assess  the 
value  of  the  IVCRTS  in  teaching  competency  recognition. 

After  investigating  the  OECicS  request  and  coordinating  with  the  Army  Com¬ 
municative  Technology  Office  (ACTO),  TDI  agreed  to  fund  and  monitor  a  formal 
evaluation  of  the  IVCRTS.  In  addition  to  an  interest  in  the  competency 
recognition  issue,  TDI  was  also  interested  in  learning  if  the  IVCRTS  has  potential 
for  application  in  training  other  types  of  skills,  particularly  if  it  could  be  used 
with  a  disk  system  to  allow  for  "real-time"  interaction  capability. 
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Due  to  limitations  imposed  by  the  selection  of  videotapes  used  in  this  effort, 
only  15  of  the  IS  consulting  competencies  were  coded  for  recognition  purposes. 

OBJECTIVE 

The  objective  of  this  effort  was  for  McFann,  Gray  &  Associates,  Inc.,  (MGA)  to 
evaluate  the  effectiveness  and  efficiency  of  the  IVCRTS  in  training  trainers  at 
OEC&S  to  recognize  15  of  the  consulting  competencies.  In  addition,  the 
usability  and  acceptability  of  the  inputting  and  coding  system  used  by  course 
developers  and  the  individual  training  management  aspects  of  the  IVCRTS  were 
evaluated. 

THE  IVCRTS  IMPLEMENTATION 

The  basic  hardware  in  the  system  includes  an  IBM  Personal  Computer,  an  ITS 
2000  Video  Controller  and  an  Electrohome  Hi-Res  RGB  Video  Monitor.  Inter¬ 
actions  between  the  IBM  computer,  the  ITS  controller  and  the  ITS  software 
enable  the  user  to  interact  with  audio- video  presentations  originating  from  a 
videotape  player.  The  15  competencies  are: 


I. 

Functional  Knowledge 

9. 

Aware  of  Self  Impact 

2. 

Self  Confidence 

10. 

Seeks  Multiple  Perspectives 

3. 

Low  Fear  of  Rejection 

11. 

Causal  Thinking 

4. 

Establishes  Professional  Rapport 

12. 

Identifies  Key  Themes 

5. 

Concern  for  Clarity 

13. 

Problem  Focused  Procedures 

6. 

Seeks  User  Input 

14. 

5eizes  Opportunities 

7. 

Uses  Influence  Strategies 

15. 

Results  Orientation 

8. 

Communicates  Clearly 

The 

ITS  software  is  designed  to  facilitate  learning  by  requiring  users  to  connect 

a  set  of  terms  or  concepts  with  visual 

images, 

i.e.,  by  requiring  a  learner  to 

become  skilled  at  identifying  something  that  is  presented  in  the  visual  mode. 
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The  IVCRTS  package  consists  of  three  major  components: 

•  The  Inputting  and  Coding  System.  This  system  was  used  by  course 
developers  (coders)  to  code  15  competencies  onto  3  different 
videotapes.  (The  videotapes  selected  for  this  purpose  did  not  contain 
segments  suitable  for  coding  the  other  three  competencies  considered 
trainable  in  the  course.)  When  a  competency  was  being  displayed  on  the 
videotape,  the  coder  indicated  which  competency  was  being  displayed, 
where  the  segment  began  and  ended,  and  the  strength  of  the  display. 
The  coder  used  a  keyboard  to  interact  with  the  system. 

•  The  Training  System.  Once  the  tapes  were  coded,  they  were  then  used 
to  train  the  trainers.  The  trainers  used  a  light  pen  to  interact  with  the 
system.  They  chose  from  a  map  of  feature  options  which  included: 

1.  Introduction  to  the  OE  Model: 

This  feature  provides,  in  text  form,  an  explanation  of  the 
competency  model,  definitions  for  each  competency,  and 
behavioral  indicators  for  each  competency. 

2.  "Name  It"  or  "Find  It": 


This  is  the  principal  training  feature  of  the  system.  Three 
videotapes  are  available  which  portray  an  OE  Consultant 
interacting  with  clients  during  each  of  three  phases  of  the 
consulting  process  -  Assessment,  Planning,  and  Evaluation.  Each 
tape  is  coded  to  display  7  or  8  of  the  15  competencies.  The  user  is 
required  to  identify  the  competencies  displayed  by  participating  in 
a  "Name  It'  exercise  or  a  "Find  It"  exercise.  In  the  "Name  It" 
exercise,  an  audio  cue  indicates  that  a  competency  is  being 
displayed.  The  user  then  indicates  which  specific  competency  is 
being  displayed.  In  the  "Find  It"  exercise,  the  user  is  required  to 
identify  the  correct  competency  without  benefit  from  the  audio 
cue. 
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3.  Reliability  Scores: 

\  This  feature  provides  the  user  with  a  summary  of  their  progress.  A 

histogram  of  score  data  is  presented  for  each  competency. 

4.  Tutor: 

This  feature  allows  a  user  to  review  the  competencies  presented  on 
each  tape.  It  indicates  the  number  of  times  each  competency  is 
|  presented,  and  provides  definitions  and  supporting  behavioral 

indicators  of  each.  It  will  also  replay  the  videotape  segment  on 
which  each  competency  is  displayed. 

5.  Rundown: 

This  feature  records  the  instances  in  which  the  user  consistently 
confuses  one  competency  with  another  and  provides  explanations, 
to  assist  in  eliminating  that  confusion. 

•  The  Training  Management  System.  The  IVCRTS  also  includes  an 
individual  training  management  component.  Each  trainer  has  their  own 
floppy  disk  on  which  their  learning  is  tracked.  The  computer 
summarizes  a  trainer's  progress  by  showing  a  histogram  of  their  scores. 
This  is  computed  by  dividing  the  number  of  successful  trials  by  the 
number  of  times  they  have  been  exposed  to  each  competency.  This  can 
be  displayed  showing  results  across  practice  sessions  or  by  each 
competency. 

SPECIFIC  EVALUATION  ISSUES 

MGA  researchers  evaluated  each  of  the  above  components  of  the  IVCRTS 
separately  as  the  evaluation  issues  are  somewhat  different.  The  evaluation 
issues  are  briefly  summarized  below. 
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A.  Inputting  and  Coding  System 


1.  Ease  of  use. 

This  issue  addressed  whether  the  inputting  and  coding  system  was 
understandable  to  course  developers,  and  whether  they  learned  to  use  it 
easily. 

2.  Program  flexibility. 

This  issue  was  concerned  with  whether  changes  in  coding  could  be  made 
easily  and  quickly,  and  whether  the  program  was  flexible  enough  to 
code  sufficient  competencies  to  reflect  a  real-life  situation. 

3.  Acceptance  by  course  developers. 

This  issue  involves  how  positively  the  system  was  received  by  course 
developers. 

4.  Reliability. 

The  reliability  of  the  hardware  and  software  was  evaluated  by  amount 
of  downtime,  whether  the  software  followed  a  logical  sequence,  and 
whether  there  were  any  technical  problems  with  the  system. 

B.  Training  System 

1.  Effectiveness. 

This  issue  addressed  the  question,  "Does  the  IVCRTS  train  the  trainers 
to  recognize  competencies?"  OEC&S  set  a  criterion  that  70%  of  the 
trainers  must  recognize  70%  of  the  competencies  in  order  for  the 
system  to  be  considered  effective.  In  addition,  the  extent  or  degree  of 
improvement  was  to  be  assessed,  as  well  as  more  subjective  judgments 
of  effectiveness  as  measured  by  questionnaire  and  interview  items. 
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2.  Efficiency. 

The  efficiency  of  the  training  system  was  viewed  primarily  in  terms  of 
times  "How  long  does  it  take  for  an  individual  to  reach  criterion?"  And 
opinions  were  solicited  regarding  how  this  system  compares  to  other 
potential  training  methods. 

3.  User  acceptance. 

General  reactions  to  the  system  were  obtained.  In  addition,  specific 
issues  such  as  agreement  with  the  coding  and  reactions  to  the  game 
formats  were  also  addressed. 

4.  Reliability. 

As  with  the  inputting  and  coding  system,  the  amount  of  downtime, 
technical  problems,  and  logical  problems  in  the  software  were  used  to 
assess  the  system's  reliability. 

Training  Management  System 

1.  Cost  benefit. 


The  cost  benefits  of  the  system  were  evaluated  in  terms  of  time,  and 
the  effectiveness  with  which  the  system  can  adequately  assess  an 
individual's  skill  level  at  competency  recognition  were  assessed. 

2.  Effectiveness. 


The  effectiveness  of  the  training  management  system  was  evaluated 
using  such  criteria  as  access  to  the  system  and  whether  the  feedback 
regarding  performance  was  adequate. 


REPORT  ORGANIZATION 


The  methodology  followed  in  evaluating  the  IVCRTS  is  in  the  following  section, 
Section  II.  It  explains  how  the  data  were  collected,  provides  a  discussion  of  how 
the  instruments  relate  to  the  evaluation  criteria,  and  describes  the  data  analysis 
procedures.  Section  III  describes  the  sample  and  treatment.  Section  IV  is  a 
discussion  of  the  study  and  findings.  Finally,  Section  V  provides  conclusions  and 
recommendations.  The  appendices  contain  the  data  collection  instruments  with 
a  summary  of  the  data  on  each. 


SECTION  Os  EVALUATION  DESIGN 


OVERALL  DESIGN 

This  evaluation  of  the  IVCRTS  was  designed  to  provide  feedback  regarding 
whether  or  not  the  objective  of  training  OECicS  trainers  to  recognize  OE 
Consultant  competencies  was  met.  The  design,  methodology,  and  data  collection 
instruments  were  developed  by  MGA  and  approved  by  representatives  of  OEC<5cS 
and  TDI.  Details  presented  below  follow  the  sequencing  contained  in  the 
approved  Test  and  Evaluation  Plan  (A  Test  and  Evaluation  Plan  for  the 
Interactive  Video  Competency  Recognition  System,  Johnson,  C.A.,  Ph.D., 
December  1982). 

INSTRUMENTS 

Three  categories  of  data  sources  were  planned  for  use  in  evaluating  the  IVCRTS. 

1.  First,  the  IVCRTS  itself  was  to  collect  and  analyze  data  on  student 
performance  including  the  proportion  of  correctly  identified 
competencies,  the  number  of  trials  per  competency  to  reach  criterion, 
and  an  analysis  of  consistent  patterns  of  confusion  in  competencies. 

OEC&S  manually  extracted  the  original  set  of  such  data  from  the 
IVCRTS.  The  researcher's  review  of  this  data  set  determined  that 
additional  data  were  needed,  and  OEC&S  was  asked  to  provide  the  data 
of  the  type  and  format  specified  in  the  collection  plan  (page  12,  Test 
and  Evaluation  Plan).  An  extension  of  the  period  of  performance  was 
allowed  by  TDI  for  this  purpose.  OEC&S  subsequently  obtained  a 
software  program  for  use  in  extracting  the  desired  data  from  the 
system.  The  data  set  produced  via  that  program  is  contained  in 
Appendix  A. 

As  can  be  seen  in  a  review  of  that  data  set,  it  still  does  not  provide  the 
information  specified  in  the  collection  plan,  i.e.,  the  mean  proportion  of 
trainers  who  successfully  recognized  each  competency  and  the  mean 


number  of  trials  per  competency  to  reach  the  established  criterion  (70% 
correct  recognition).  Nor  were  data  available  regarding  the 
competencies  which  tend  to  be  confused  with  each  other. 

2.  Second,  structured  interviews  were  conducted.  Separate  interview 
outlines  were  developed  and  interviews  conducted  for  the  following 
groups  and  number  of  people: 

a.  Two  coders/course  developer; 

b.  Twelve  trainers;  and 

c.  The  Director  of  Training,  the  Deputy  Director  of  Training,  and 
the  one  Division  Chief. 

The  interview  outlines  and  the  aggregated  data  for  each  group  are  at 
Appendices  B,  C,  and  D,  respectively. 

3.  Finally,  two  instruments  were  used  to  collect  information  from  the 
IVCRTS  users  while  they  were  using  the  system  (Problem/Survey  Log) 
and  at  the  end  of  the  evaluation  period  (IVCRTS  Evaluation 
Questionnaire).  These  instruments,  with  the  aggregated  data,  are  at 
Appendices  E  and  F. 

The  relationship  between  the  evaluation  criteria  and  the  data  collection  instru¬ 
ments  is  displayed  in  Tables  1  through  3.  The  tables  show  the  specific  items  in 
each  relevant  instrument  that  provide  information  regarding  each  criterion. 
Table  1  addresses  the  evaluation  of  the  Inputting  and  Coding  System.  The 
specific  data  sources  for  evaluation  of  the  Training  System  are  shown  in  Table  2, 
and  those  to  evaluate  the  Training  Management  System  are  in  Table  3. 


EVALUATION  CRITERIA 


Numbers  refer  to  items  in  instruments 


EVALUATION  CRITERIA 


Numbers  refer  to  items  in  instruments 


TABLE  3 


DATA  ANALYSIS 


IVCRTS  Data 


The  data  collected  by  the  computer  system  are  in  two  forms:  1)  The  percent  of 
competency  presentations  correctly  identified  by  Training  Session  and  by 
Trainer(s);  and  2)  the  percent  correct  recognitions  for  the  total  number  of 
presentations  for  each  competency.  In  each  form,  these  data  are  provided 
separately  for  the  "Name  It"  exercise  and  the  "Find  It"  exercise.  No  data  or 
computations  were  provided  concerning  the  specific  nature  of  each  training 
session,  i.e.,  the  specific  competencies  presented  and  the  frequency  of  presen¬ 
tation  within  a  given  training  session.  The  most  useful  analysis  made  of  the 
IVCRTS  data  involved  computing  the  mean  percentage  of  correctly  identified 
presentations  by  session  and  trainer(s)  and  percentage  of  trainers  with  a  mean  of 
70  or  more  percent  correct  competency  identification  across  sessions. 

Structured  Interviews 


A  content  analysis  was  performed  on  the  data  collected  in  the  three  sets  of 
interviews  (Coders/Course  Developers,  Trainers,  and  Managers).  As  depicted  in 
Tables  1  through  3,  aggregated  data  from  relevant  source  items  were  used  to 
evaluate  each  of  the  IVCRTS  subsystems,  i.e,  Inputting  and  Coding,  Training,  and 
Training  Manager.  Appropriate  response  categories  were  developed  and  fre¬ 
quencies  and  percent  of  responses  in  each  category  were  tabulated  and  recorded 
for  the  data  from  the  12  trainer  interviews. 

Problem  Survey  Log 

Responses  to  the  Problem/Survey  Log  are  reported  as  frequencies  and  per¬ 
centages.  The  data  were  also  examined  as  a  function  of  experience  on  the 
IVCRTS,  i.e.,  the  number  of  times  individuals  used  the  system.  Two  sub-groups 
were  used  for  this  purpose:  1)  a  group  of  17  respondents  indicating  one-time  use, 
and  2)  a  group  of  7  respondents  indicating  multiple  use,  i.e.,  3  or  more  times. 
Aggregated  data  for  these  two  groups  are  presented  as  separate  enclosures  to 
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Appendix  E.  Additionally,  written  comments  were  examined,  appropriate 
categories  were  developed,  and  frequencies  and  percent  of  response  in  each 
category  were  tabulated  and  recorded. 


1VCRTS  Evaluation  Questionnaire 

This  questionnaire  contains  Likert  scale  response  items.  The  frequency  and 
percentage  of  responses  for  each  scale  point  is  reported  along  with  the  mean, 
variance,  and  standard  deviation  of  each  item.  The  same  examination  was  made 
on  the  data  from  three  sub-groups:  1)  13  military,  2)  3  civilian,  and  3)  7 
individuals  who  indicated  that  they  had  one  year  or  more  experience  as  both  a 
trainer  and  a  consultant.  Aggregated  data  for  these  sub-groups  are  presented  as 
separate  enclosures  to  Appendix  F. 


SECTION  ms  SAMPLE  AND  TREATMENT 


SAMPLE 

The  IVCRTS  was  used  to  train  20  trainers  to  recognize  the  set  of  competencies. 
Fourteen  of  the  20  worked  in  pairs.  In  addition  to  data  collected  from  this  group 
of  trainers,  9  other  individuals  provided  data  used  in  the  evaluation  effort:  2 
course  developers  who  coded  the  competencies  were  interviewed;  4  course 
developers  who  used  the  IVCRTS  completed  the  Problem  Survey/Log  and  IVCRTS 
Evaluation  Questionnaire;  and  3  training  managers  were  interviewed  (the 
Director  of  Training,  Deputy  Director  of  Training,  and  a  Division  Chief).  Table  4 
shows  the  number  and  types  of  individuals  in  the  sample  and  the  data  collection 
methods  used  for  each. 


TABLE  4:  SAMPLE  DESCRIPTION 


COLLECTION  METHOD 

SOURCE 

NUMBER 

IVCRTS  Program 

Trainers 

e 

20 

(14  in  Pairs) 

Problem  Survey  Log 

Trainers 

Course  Developers 

20 

4 

IVCRTS  Evaluation  Questionnaire 

Trainers 

Course  Developers 

12* 

4 

Trainers 

12* 

Structured  Interviews 

Managers 

3 

Coders 

2 

*A  subset  of  the  20  trainers  who  used  the  IVCRTS. 
TREATMENT 


General 


The  20  trainers  assigned  to  the  Training  Directorate  were  informed  through 
normal  command  channels  that  they  would  participate  in  an  evaluation  of  the 
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IVCRTS.  They  were  instructed  to  use  the  system  at  their  convenience  during  the 
evaluation  period  (December  to  March)  at  one  of  two  locations  (the  OEC&S 
Library  Learning  Center  and  an  office  in  the  Training  Development  (TD) 
Directorate).  Both  locations  were  suitably  equipped  with  IVCRTS  equipment  and 
written  instructions.  All  persons  who  used  the  IVCRTS  received  a  thorough 
briefing  on  the  the  system  during  an  initial  orientation  session.  Written 
instructions  were  available  for  subsequent  use,  and  TD  personnel  were  on-call  to 
answer  questions  and  provide  assistance. 

In  all  cases,  trainers  were  told  they  were  expected  to  reach  the  established 
criterion  (70%  correct  recognition  of  70%  of  the  15  competencies  displayed). 
Late  in  the  evaluation  period,  OECicS  training  managers  learned  that  only  a  few 
of  the  trainers  were  actually  using  the  IVCRTS.  Consequently,  the  Director  of 
Training  issued  instructions  which  resulted  in  each  trainer  being  scheduled  to  use 
the  IVCRTS  when  the  trainer  was  not  scheduled  for  instructor  duties  in  the 
classroom. 

Videotapes 

Competencies  were  coded  onto  three  pre-existing  videotapes.  Each  contains  15 
to  20  minutes  of  continuous  action  and  was  originally  produced  for  use  in  the 
TRADOC  school  system  to  introduce  and  explain  the  OE  process  to  service 
school  students.  On  the  tapes,  professional  actors  portray  an  Army  OE 
consultant  interacting  with  clients  in  a  military  setting.  In  addition  to  the 
client-consultant  interactions,  a  narrator  periodically  explains  various  aspects  of 
the  consultant's  activities.  Each  tape  has  as  its  principal  focus  one  of  three 
steps  in  the  four -step  OE  process.  Thus,  one  tape  explains  and  demonstrates 
activities  associated  with  the  Assessment  step,  one  tape  deals  with  the  Planning 
step,  and  one  deals  with  the  Evaluation  step.  The  coders  located  tape  segments 
they  believed  to  be  accurate  representations  of  the  15  competencies  under 
consideration  and  coded  the  tapes  accordingly.  Depending  on  the  tape  used  and 
the  exercise  option  selected  (i.e,  "Name  It  or  Find  It"),  the  number  of 
competencies  available  for  display  and  the  frequency  with  which  they  are 
presented  varies  (see  Table  5). 
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TABLE  5:  FREQUENCY  OF  COMPETENCY  PRESENTATIONS 


Competency 

Assessment 

Tape 

(Frequency) 

Planning 

Tape 

(Frequency) 

Evaluation 

Tape 

(Frequency) 

1.  Functional  Knowledge 

1 

1 

2.  Self  Confidence 

1 

1 

3.  Low  Fear  of  Rejection 

2 

4.  Establishes 

Professional  Rapport 

1 

5.  Concern  for  Clarity 

1 

4 

2 

6.  Seeks  User  Input 

2 

7.  Uses  Influence 

Strategies 

1 

8.  Communicates  Clearly 

1 

9.  Aware  of  Self  Impact 

1 

10.  Seeks  Multiple 
Perspectives 

1 

2 

11.  Causal  Thinking 

I 

12.  Identifies  Key 

Themes 

I 

2 

1 

13.  Problem  Focused 
Procedures 

2 

14.  Seizes  Opportunities 

1 

15.  Results  Orientation 

1 

1 

IVCRTS  Use  Options 

i l. » _ : _  _ _ _ _  :  t i a.l -  ... _ -i : i i •  /t..  •  c t i  
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•  Introduction  to  the  OE  Model 

•  Exercise  Play:  "Name  It"  or  "Find  It" 

•  Reliability  Scores 

•  Tutor 

•  Rundown 

Ideally,  a  user  would  begin  with  the  Introduction  option  to  learn  as  much  as 
possible  about  the  competencies  available  for  display  and  the  behavioral  indi¬ 
cators  for  each.  When  ready,  a  user  would  then  select  one  of  the  three  available 
tapes  (i.e.,  Assessment,  Planning,  or  Evaluation)  and  one  of  the  two  exercise 
modes  (i.e.,  "Name  It"  or  "Find  It"). 

The  "Name  It"  exercise  is  considered  the  easier  of  the  two  because  it  provides  an 
audio  cue  to  indicate  when  a  competency  presentation  segment  begins.  It  also 
stops  at  the  end  of  each  such  segment  and  provides  the  user  with  three  options: 
1)  Replay  the  Segment;  2)  Reliability  Score;  and  3)  Name  the  Competency. 

An  option  within  each  exercise  allows  users  to  indicate  that  they  will  concen¬ 
trate  on  a  subset  of  the  competencies  coded  on  the  tape  in  use  (i.e.,  from  three 
to  seven  or  eight).  Thus,  in  the  case  of  "Name  It,"  if  a  user  indicates  interest  in 
only  three  specific  competencies  on  a  given  tape,  the  audio  will  cue  only  those 
segments  of  the  tape,  and  the  list  of  possibles  for  identification  will  contain  only 
those  competencies  identified. 

"Find  It"  is  considered  more  difficult  in  that  there  is  no  audio  cue  to  indicate 
which  tape  segment  to  attend  to.  Nor  does  the  system,  in  this  exercise  mode, 
automatically  provide  a  selection  list  at  the  end  of  each  presentation  segment. 
The  user  must  touch  the  screen  during  the  presentation  segment  to  stop  the  play 
and  get  a  selection  list  from  which  to  choose  the  correct  competency.  If  the 
user  does  not  detect  a  presentation  and  touch  the  screen  during  the  appropriate 
segment,  the  system  stops  play,  informs  the  user  of  a  missed  presentation,  and 
provides  the  following  options:  continue;  return  to  review  the  segment;  identify 
the  competency;  or  check  reliability  score.  If  the  user  elects  to  continue  or 
incorrectly  identifies  the  competency,  a  missed  identification  is  scored  and 
recorded. 


Given  this  difference  in  levels  of  difficulty  between  the  two  exercises,  the 
trainers  were  advised  to  work  through  the  tapes  using  the  "Name  It"  exercise 
option  before  attempting  the  "Find  It"  exercise.  And,  because  the  Planning  tape 
was  thought  to  be  programmed  for  use  in  the  "Find  It"  mode  only,  the  trainers 
were  further  advised  to  use  the  Assessment  and/or  Evaluation  tape  before  the 
Planning  tape.  (Recent  information  from  OECicS  indicates  that  the  Planning 
tape  used  at  the  Library  location  was  mistakenly  programmed  for  use  in  both  the 
"Name  It"  and  "Find  It"  modes.) 

The  Tutor  feature  is  available  for  users  who  have  trouble  identifying  any  of  the 
competency  presentations.  An  important  feature  of  the  tutor  option  allows  users 
to  review  any  competency  they  are  having  trouble  with. 

The  Reliability  Score  feature  simply  provides  the  user  with  a  cumulative  record 
of  performance.  This  is  the  feature  the  trainers  used  to  determine  if/when  they 
reached  the  established  criterion. 

The  Rundown  feature  would  be  helpful  to  the  user  who  consistently  confuses  one  ! 

or  more  competencies  with  others.  In  this  instance,  the  feature  would  display 
data  only  when  a  competency  was  confused  with  another  competency  at  least 

three  times.  , 

I 

Training  Session 

The  individual  session  is  the  basic  activity  from  which  all  IVCRTS  score  data  1 

originate.  It  consists  of:  \ 

<1 

1.  A  user  (or  in  many  cases,  a  pair  of  users)  signing  on  the  system;  ] 

\ 

t 

2.  Participating  from  start  to  finish  in  one  of  the  two  exercises  j 

("Name  It"  or  "Find  It");  ! 


3.  Using  one  of  three  tapes  ("Assessment,"  "Planning,"  or 
"Evaluation"); 


4.  Selecting  from  three  to  eight  specific  competencies  available  on 
the  tape;  and 

5.  Signing  off  the  system. 

Thus,  as  depicted  in  Table  6,  there  are  a  total  of  34  activity  combinations 
possible  for  a  given  training  session.  A  single  training  session,  therefore,  would 
involve  some  subset  of  the  total  number  of  possible  combinations  depending  on: 
1)  the  tape  used,  2)  the  exercise  option  selected,  and  3)  the  number  of 
competencies  selected  from  those  available. 

The  IVCRTS  software  did  not  indicate  which  tape  was  used  or  which  compe¬ 
tencies  were  presented  within  each  training  session.  This  lack  of  information 
combined  with  the  number  of  combinations  possible  made  it  difficult  to  interpret 
the  IVCRTS  performance  data.  These  data  are  discussed  in  Section  IV. 


TABLE  6:  COMBINATIONS  POSSIBLE 


TAPE 

EXERCISES  AND  PRESENTATION 
OPTIONS 

COMBINATIONS 

POSSIBLE 

NAME  IT 

FIND  IT 

Assessment 

3  competencies 

4  competencies 

5  competencies 

6  competencies 

7  competencies 

3  competencies 

4  competencies 

5  competencies 

6  competencies 

7  competencies 

10 

♦Planning 

3  competencies 

4  competencies 

5  competencies 

6  competencies 

7  competencies 

8  competencies 

3  competencies 

4  competencies 

5  competencies 

6  competencies 

7  competencies 

8  competencies 

12 

Evaluation 

3  competencies 

4  competencies 

5  competencies 

6  competencies 

7  competencies 

8  competencies 

3  competencies 

4  competencies 

3  competencies 

6  competencies 

7  competencies 

8  competencies 

12 

♦This  set  of  options  was  available  only  at  the  Library  Learning  Center  site. 


Procedure 


After  being  introduced  to  the  system  by  verbal  briefing  and  given  a  set  of 
written  instructions,  the  trainer  or  pair  of  trainers,  were  free  to  use  the  IVCRTS 
as  they  pleased.  (They  were  expected  to  work  as  previously  instructed,  i.e., 
using  "Name  It"  before  "Find  It"  and  Assessment  and/or  Evaluation  before 
Planning.)  No  data  were  collected  for  each  session  to  indicate  which  tape  was 
used,  which  exercise  was  selected,  or  which  set  or  subset  of  available  compe¬ 
tencies  was  drilled. 

The  trainer(s)  were  told  to  work  on  the  system  until  they  had  at  least  a  70% 
reliability  score  for  each  exercise  option  available  on  each  of  the  three  tapes. 
That  is,  each  trainer,  or  pair  of  trainers,  was  expected  to  record  at  least  one 
succeessful  session  (a  70%  reliability  score  for  all  competencies  available  for 
display)  for  each  of  the  following  exercise  options: 


Tape 

Options 

Available 

Cumulative 
Number  of 
Options 

•  Assessment  Tape 

Name  It 

1 

Find  It 

2 

•  Planning  Tape 

Find  It 

3 

•  Evaluation  Tape 

Name  It 

4 

Find  It 

5 

This  requires  a  minimum  of  five  sessions  with  a  reliability  score  of  70%  for  each 
trainer  or  pair  of  trainers  (i.e.,  two  successful  sessions  on  "Name  It"  and 
three  successful  sessions  on  "Find  It"). 

Because  complete  anonymity  was  desired,  user  disks  were  coded  to  prevent 
monitoring  of  individual  programs.  Trainers  were  trusted  to  continue  to  use  the 
IVCRTS  until  they  recorded  the  five  successful  sessions  just  described. 


Each  trainer  was  to  complete  a  Problem  Survey/Log  after  the  first,  fourth,  and 
last  session,  and  at  any  other  time  desired.  Based  on  the  number  of  sessions 
recorded,  this  could  have  generated  a  minimum  of  54  responses.  Forty-four 
percent  (or  24)  of  this  number  were  received. 

The  Evaluation  Questionnaire  was  to  be  completed  by  all  20  trainers  upon 
completion  of  the  evaluation  effort.  Twelve  were  received  for  a  response  rate 
of  60  percent.  Twelve  trainers  were  interviewed  by  MGA  project  persons  after 
they  participated  in  the  1VCRTS  training. 


SECTION  IV:  ENDINGS  AND  DISCUSSION 


GENERAL 

Evaluation  Issues 

This  discussion  will  focus  on  the  specific  evaluation  issues  described  in  Section  I 
of  this  report.  These  issues  are  listed  below  as  they  relate  to  the  three 
components  of  the  IVCRTS: 

Issue 

Ease  of  Use 
Program  Flexibility 
Acceptance  by  Course  Developers 
Reliability 

Effectiveness 
Efficiency 
User  Acceptance 
Reliability 

Cost  Benefit 
Effectiveness 

Data  Summaries 

The  complete  data  on  which  this  discussion  is  based  are  contained  in  Appendices 
A  through  F.  Summary  tables  from  the  three  largest  data  sets  are  provided  here 
to  assist  in  following  the  discussion.  They  are: 

1.  IVCRTS  Data 

Table  7  is  a  display  of  trainer  performance  scores  by  training  session. 
The  figures  in  the  "Session”  columns  represent  the  percent  of  correctly 
identified  competencies  for  each  session  of  the  "Name  It"  and  "Find  It" 
exercises.  No  data  were  provided  to  indicate  which  competencies  were 
actually  presented  and  with  what  frequency. 
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TABLE  7:  PERCENT  COMPETENCY  PRESENTATIONS  CORRECT  BY  SESSION  AND  TRAINER(S) 


Trainer 

1 

Name  It 
Find  It 

2 

Name  It 
Find  It 

3 

Name  It 
Find  It 

4 

Name  It 
Find  It 

5 

Name  It 
Find  It 

6 

Name  It 
Find  It 

7 

Name  It 
Find  It 

S 

Name  It 
Find  It 

9 

Name  It 
Find  It 

10 

Name  It 
Find  It 

11 

Name  It 
Find  It 

12 

Name  It 
Find  It 

13 

Name  It 
Find  It 

90 

SO 

3 

Mean* 


50 

80 

_ 

_ 

_ 

-  72 

33 

“ 

- 

- 

- 

— :L. 58 

♦Percent  of  trainers  with  mean  session  scores  at  or  above  70:  Name  It  -  54;  Find 
It  -  31 


Information  from  Table  5  (page  17)  indicates  that  only  7  or  8  of  the  15 
competencies  are  available  for  presentation  during  a  single  session. 
Furthermore,  one  of  the  use  options  makes  it  possible  for  a  user  to 
further  restrict  the  number  of  competencies  presented  during  a  session 
to  a  minimum  of  3. 

Thus,  a  single  session  provides  from  20  to  53%  of  the  15  competencies, 
which  means  that  a  minimum  of  3  sessions  are  required  for  exposure  to 
all  15  competencies  tf  all  tapes  and  all  exercise  options  are  chosen. 

Furthermore,  3  of  the  competencies,  or  20%,  are  available  only  on  the 
Planning  tape,  which,  for  the  most  part,  was  used  only  with  the  "Find 
It"  exercise. 

Regarding  frequency  of  presentations,  the  Assessment  and  Evaluation 
tapes  (available  with  "Name  It"  and  "Find  It"  exercises)  presented  all 
but  2  of  12  competencies  only  once  during  a  single  training  session. 
Thus,  learning  gains  would  have  to  be  tracked  across  sessions  rather 
than  within  sessions. 

Of  the  8  competencies  available  on  the  Planning  tape  (for  use  with  the 
"Find  It"  exercise  only),  one  is  presented  4  times,  4  are  presented  twice, 
and  3  are  presented  only  once. 

Table  8  is  a  display  of  the  mean  performance  scores  for  all  trainers  by 
competency.  Figures  in  the  "Presentations"  column  show  the  total 
number  of  times  a  given  competency  was  presented  in  the  "Name  It" 
and  the  "Find  It"  exercises  for  all  sessions.  Figures  in  the 
"Recognitions"  column  show  the  percent  of  correct  recognitions  for  all 
presentations  of  the  competency. 
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TABLE  8:  PERCENT  CORRECT  RECOGNITION  BY  COMPETENCY 


Competency 

Exercise 

Option 

Total 

Number 

Presentations 

Percent 

Correct 

Recognitions 

Mean 

Functional 

Name  It 

65 

63 

Knowledge 

Find  It 

44 

66 

65 

Self 

Name  It 

37 

70 

- | 

Confidence 

Find  It 

56 

70 

70 

Low  Fear  of 

Name  It 

64 

77 

Rejection 

Find  It 

43 

79 

78 

Establishes 

Name  It 

34 

89 

Professional  Rapport 

Find  It 

23 

83 

86 

Concern  for 

Name  It 

112 

72 

Clarity 

Find  It 

175 

58 

65  | 

Seeks  User 

Name  It 

12 

75 

i 

Input 

Find  It 

70 

52 

6  u 

Uses  Influence 

Name  It 

8 

25 

Strategies 

Find  It 

34 

35 

30 

Communicates 

Name  It 

33 

61 

Clearly 

Find  It 

27 

52 

Aware  of 

Name  It 

33 

70 

Self  Impact 

Find  It 

24 

67 

69 

Seeks  Multiple 

Name  It 

40 

75 

Perspectives 

Find  It 

61 

59 

67 

Causal 

Name  It 

30 

77 

Thinking 

Find  It 

21 

95 

86 

Identifies  Key 

Name  It 

78 

S3 

Themes 

Find  It 

117 

69 

76 

Problem  Focused 

Name  It 

13 

54 

Behavior 

Find  It 

85 

48 

51 

Seizes 

Name  It 

32 

100 

Opportunities 

Find  It 

21 

95 

98 

Results 

Name  It 

38 

69 

Orientation 

Find  It 

71 

45 

55 

Mean/Standard 

Name  It 

67.30/20.50 

69 

Deviation 

Find  It 

64.90/17.68 
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2.  Problem  Survey/Log 


Table  9  provides  a  summary  comparison,  by  groups,  of  the  data 
obtained  from  the  Problem  Survey/Log.  The  number  responding  to 
each  survey  item  and  the  percentage  of  responses  to  item  options 
are  shown  for  all  data,  the  one  time  users,  and  the  multiple  users. 

3.  IVCRTS  Evaluation  Questionnaires 

Table  10  shows  the  scale,  mean  responses,  and  standard  deviation 
for  All  Data,  Military,  Experienced  Consultants/Trainers,  and 
Civilians.  For  items  1  through  10,  the  low  end  of  the  scale  is 
anchored  to  the  negative  verbal  cue,  the  high  end  to  the  positive 
cue,  and  the  midpoint  on  the  scale  is  anchored  to  the  neutral  cue. 

This  comparison  indicates  that  there  were  no  significant  differences  in  the 
responses  by  groups.  (It  is  noted,  however,  that  the  three  civilian  members  are 
generally  less  positive  in  their  responses.) 

In  the  discussion  of  these  data  that  follows,  references  are  made  to  the  All  Data 
Summary  contained  in  Table  11.  This  table  reproduces  the  questionnaire  and 
provides  the  mean  response  for  each  item  (horizontal  marker)  and  one  standard 
deviation  above  and  below  the  mean  (vertical  line). 

Interview  data  from  the  2  coders,  12  trainers,  and  3  managers  are  in  summary 
form  in  Appendices  8,  C,  and  D,  respectively. 


TABLE  9:  PROBLEM  SURVEY/LOG 
ALL  DATA,  ONE-TIME  USE,  AND  MULTIPLE  USE  COMPARISON 


One-Time  Multiple 

Survey/Log  All  Data  Use  Data  Use  Data 

Item:  N=24  N=17  N=7 


2. 

Features  Used: 

(N=22) 

(N=15) 

(N=7) 

Introduction- 

.32 

.40 

.00 

"Name-It"- 

.86 

.93 

.57 

"Find-It"- 

.64 

.47 

1.00 

Tutor- 

.09 

.07 

.14 

Reliability  Scores- 

.73 

.87 

.29 

Rundown- 

.55 

.67 

.29 

Problems  Encountered: 

(N=17) 

( N=  11) 

(N=6) 

3. 

Unsure  what  to  do 
next- 

.53 

.55 

.33 

4. 

Touched  panel  and 
nothing  happened- 

.76 

.82 

.83 

5. 

Sure  gave  right 
answer  but  it 
differed  from 
coding- 

.88 

.91 

.67 

6. 

Terminology  confusing- 

o 

o 

• 

.00 

.00 

7. 

Needed  to  ask  for 
assistance- 

.53 

.45 

.50 

8. 

Trouble  getting 
access- 

.12 

.09 

.17  1 

9.  If  told  you  were 

incorrect,  were  you 
satisfied  as  to  why? 

(N=23) 

N=16) 

(N=7) 

Yes 

.04 

0 

.14 

Somewhat 

.39 

.38 

.43 

No 

.57 

.63 

.43 

TABLE  10:  IVCRTS  EVALUATION  QUESTIONNAIRE 
ALL  DATA  AND  SUBGROUP  COMPARISON 


MEAN  RESPONSE/STANDARD  DEVIATION  BY  GROUP 


Questionnaire 

Items 

All 

N=16 

Mean 

(SD) 

Military 

N=13 

Mean 

(SD) 

Experienced 

N=7 

Mean 

(SD) 

Civilians 

N=3 

Mean 

(SD) 

1. 

Influenced  confidence  in 
recognizing  competencies. 

3.5 

3.6 

3.4 

3 

(Scale:  1  to  5) 

(.82) 

(.77) 

(.96) 

(1.0) 

2. 

Interest  while  using. 

3.5 

3.6 

3.4 

3 

(Scale:  1  to  5) 

(1.2) 

(1.3) 

(1.3) 

(1.0) 

3. 

Learning  applied  to 
classroom. 

3 

3.2 

3 

2.3 

(Scale:  1  to  5) 

(1.2) 

(1.2) 

(1.4) 

(1.2) 

4. 

IVCRTS  effectiveness  in 
training  competencies. 

3.9 

3.9 

3.7 

3.6 

(Scale:  1  to  5) 

(1.3) 

(1.3) 

(1.4) 

(1.5) 

5. 

Like  to  see  technology  used 
for  other  OEC&S  training. 

3.8 

3.8 

3.4 

3.3 

(Scale:  1  to  5) 

(1.5) 

(1.4) 

(1.7) 

(2.1) 

6. 

Accurately  simulated 

OE  consultant's  job. 

3.3 

3.2 

3.1 

3.3 

(Scale:  1  to  5) 

(1.4) 

(1.5) 

(1.5) 

(1.2) 

7. 

Availability  for  use. 

2.9 

2.8 

2.9 

3. 

(Scale:  1  to  3) 

(.34) 

(.36) 

(.38) 

(.00) 

8. 

Adequacy  of  performance 
score  display. 

2.7 

2.8 

2.7 

2.3 

(Scale:  1  to  4) 

(.95) 

(.93) 

(1.1) 

(1.2) 

9. 

Would  recommend  to  train 
recognition  skills. 

3 

3 

2.7 

3 

(Scale:  1  to  4) 

(1.1) 

(1.2) 

(1.3) 

(1.0) 

10. 

Increased  frequency  of 
using  competency  language. 

1.6 

1.6 

1.6 

1.3 

(Scale:  1  to  3) 

(.73) 

(.77) 

(.79) 

(.58) 
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TABLE  lit  IVCRTS  EVALUATION  QUESTIONNAIRE  BY  ITEM  RESPONSE  -  ALL  DATA 

(N=16) 


1.  How  did  use  of  the  IVCRTS  influence  your  confidence  in  recognizing  the 
competencies? 


Greatly  increased  my  confidence. 
Increased  my  confidence. 

Did  not  change  my  confidence. 
Decreased  my  confidence. 

Greatly  decreased  my  confidence. 


2.  How  interested  were  you  while  using  the  IVCRTS? 

•  Very  interested. 

•  Interested. 

•  Neutral. 

•  Bored. 

•  Very  bored. 


3.  How  much  of  what  you  learned  using  the  IVCRTS  could  be  applied  in  the 
classroom? 

Almost  ail  of  what  I  learned. 

Most  of  what  I  learned. 

Not  sure. 

Little  of  what  I  learned. 

None  of  what  I  learned. 


4.  How  effective  do  you  think  the  IVCRTS  is  in  training  competency  recog¬ 
nition? 


Very  effective. 
Somewhat  effective. 
Undecided. 

Somewhat  ineffective. 
Very  ineffective. 


5.  Would  you  like  to  see  IVCRTS-type  technology  used  for  additional  training 
(for  students  or  trainers)  at  OEC&S? 

•  Definitely. 

•  Probably. 

•  Undecided. 

•  Not  particularly. 

•  Definitely  no. 


TABLE  11:  (Continued) 


6.  How  accurately  do  you  think  the  videotapes  simulate  the  OE  Consultant's 
job? 


Very  poorly. 
Not  too  well. 
Undecided. 
Okay. 

Very  well. 


7.  Did  you  have  any  problem  being  able  to  use  the  1VCRTS  when  you  wanted  to 
(i.e.,  due  to  location,  hours  available,  etc.)? 


Never. 
Sometimes. 
Almost  always. 


8.  How  adequate  is  the  score  display  provided  by  the  IVCRTS  regarding  your 
performance? 


Superb. 

Good. 

Adequate. 

Poor. 


9.  Would  you  recommend  using  IVCRTS  technology  to  a  friend  who  was  looking 
for  a  way  to  train  recognition  skills? 

H*  Yes,  very  highly. 

•  Yes,  with  a  few  reservations. 

•  Maybe,  depending  on  certain  factors. 

•  No,  not  at  all. 


10.  How  much  more  frequently  do  you  use  competency  "language"  now  com¬ 
pared  to  before  you  used  the  IVCRTS? 


Not  at  all. 

Slightly  more  frequently. 
Much  more  frequently.  ■ 
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TABLE  11:  (Continued) 


11.  There  are  many  potential  methods  of  training  competency  recognition.  In 
your  opinion,  how  would  each  of  the  following  methods  compare  to  the 
IVCRTS  in  terms  of  effectiveness.  Place  a  checkmark  in  the  appropriate 
column.  (Shown  by  percent  responses.) 

Less  Effective  than  IVCRTS  More  Effective  than  IVCRTS 


Written  programmed  learning 

.81 

Structured  Experiences 

.75 

Self  study  tapes 

.80 

Role  playing 

.73 

TV  tapes 

.80 

Case  Study 

.63 

Case  study 

.38 

TV  Tapes 

.20 

Role  playing 

.27 

Self  study  tapes 

.20 

Structured  Experiences 

.25 

Written  programmed  learning 

.  19 

12.  How  long  have  you  been  a  trainer  at  OECicS? 

•  Over  4  years 

•  2-3  years 

•  1-2  years 

•  Less  than  1  year 

•  No  experience 


13.  Are  you: 

•  .81  (13)  Military 

•  .19  (3)  Civilian 


14.  Have  you  had  experience  doing  O.E./O.D.  consulting? 
If  so,  how  long? 

•  Over  5  years 

•  3-4years 

•  2-3  years 

•  1-2  years 

•  Little  experience 

•  No  experience 
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INPUTTING  AND  CODING  SYSTEM 


Two  members  of  the  OEC&S  Training  Development  staff  served  as  coders  for  the 
IVCRTS.  They  usually  worked  independently  and  served  as  a  cross-check  on  each 
other's  coding. 

Ease  of  Use 


The  majority  of  the  coder  comments  indicated  that  the  inputting  and  coding 
system  is  quite  easy  to  use.  The  system  was  considered  to  be  "very  user 
friendly"  and  did  not  overload  the  coders  with  information.  They  both  found  it 
easy  to  locate  specific  segments  on  the  video  by  stopping  the  video  at  the 
appropriate  place  or  by  locating  a  segment  with  the  frame  number.  Two  persons 
were  considered  to  be  the  optimal  number  for  coding. 

Certain  competencies  were  much  easier  to  identify  than  others  and  some  were 
easily  confused.  Those  which  were  mentioned  as  easily  identified  were  Func¬ 
tional  Knowledge,  Establishes  Professional  Rapport,  Seizes  Opportunities,  Seeks 
Multiple  Perspectives,  Communicates  Clearly,  and  Identifies  Key  Themes.  Seeks 
User  Input  and  Concern  for  Clarity  were  easy  to  identify  but  it  was  difficult  to 
identify  the  criteria  or  indicators  of  these  competencies.  The  competencies 
which  were  difficult  to  identify  for  coding  purposes  included  Awareness  of  Self 
Impact,  Perceptual  Objectivity,  Causal  Thinking,  and  Seizes  Opportunities.  This 
group  of  competencies  seems  to  be  distinguished  from  the  easily  identified  list 
by  being  less  behaviorially  oriented.  There  was  a  tendency  to  confuse  certain 
competencies.  Concern  for  Clarity  was  sometimes  confused  with  Seeks  User 
Input  and  also  with  Results  Orientation.  Self  Confidence  and  Low  Fear  of 
Rejection  also  had  a  tendency  to  be  confused.  This  appears  to  be  more  a 
function  of  the  independence  of  the  competencies  than  the  IVCRTS  system. 

The  coders  disagreed  on  the  adequacy  of  the  coding  system  instructions.  One 
found  them  to  be  complete,  the  other  felt  that  they  were  slim  and  that  it  was 
necessary  to  read  them  carefully  to  use  the  system. 


While  they  agreed  that  there  was  a  learning  curve  which  took  place  for  the 
inputting  and  coding  system,  they  disagreed  on  the  amount  of  time  it  takes  to 
learn  the  system.  One  felt  a  formal  training  progam  of  1  to  2  weeks  would  be 
advantageous,  the  other  stated  that  it  takes  only  10  minutes  to  train  a  novice  as 
a  coder.  The  coding  system  is  complex.  It  includes  9  functions,  23  tasks  and  26 
operations  to  choose  from.  It  is  necessary  to  understand  the  operations  prior  to 
selecting  a  function  or  a  task.  However,  it  was  recognized  that  complexity  was 
necessary  in  order  to  have  a  flexible  system.  It  was  recommended  that  backup 
manual  data  be  kept  initially  on  the  coding,  so  that  if  a  mistake  was  made,  the 
information  would  not  be  lost. 

Even  after  the  coders  learned  to  use  the  system,  they  found  that  it  was  not 
possible  to  code  for  extended  periods  of  time.  Coding  is  intense  and  requires  a 
high  degree  of  alertness.  Mistakes  occur  after  working  on  the  system  for  too 
long  a  time  period. 

Program  Flexibility 

Comments  regarding  the  flexibility  of  the  inputting  and  coding  system  were 
mixed.  The  coders  agreed  that  the  system  was  flexible  in  that  it  was  very  easy 
to  re-code  competencies.  There  were  initial  problems  in  modifying  the  windows 
of  a  segment,  but  after  they  learned  the  system,  they  were  able  to  make  very 
minute  adjustments. 

Initially,  there  was  some  problem  exiting  a  program.  If  no  competencies  were 
selected,  it  was  necessary  to  go  through  the  entire  video  before  exiting  the 
program.  The  coders  decided  to  handle  this  by  changing  the  program  to  require 
the  selection  of  a  minimum  of  three  competencies.  Some  tasks  will  not  allow  a 
coder  to  exit  even  if  a  wrong  choice  has  been  made.  For  example,  if  a  segment 
has  been  selected,  it  is  necessary  to  code  it  in  order  to  exit  the  program.  Then, 
if  it  has  been  incorrectly  selected,  it  is  possible  to  erase  it. 


Among  the  ways  the  coders  would  like  to  improve  flexibility  are  to  be  able  to 
add  or  delete  competencies  and  to  be  able  to  add  supporting  narrative  material 


J 

i 

to  the  program.  They  would  also  like  to  have  "Tutor"  accessed  within  the  "Name 
It"  program  in  addition  to  the  "Find  It"  program. 

The  major  disadvantage  in  terms  of  program  flexibility  is  that  the  coders  are 
restricted  to  the  given  software  which  is  expensive  to  change. 

Acceptance  by  Course  Developers 

The  1VCRTS  is  regarded  very  positively  by  course  developers.  They  are 
impressed  with  the  inputting  and  coding  system.  Other  systems  require  inputting  I 

and  coding  to  be  done  by  programmers,  using  subject  matter  expert  input,  at  the 
manufacturer's  site.  By  doing  their  own  coding,  the  OEC&S  staff  feel  a  sense  of 
ownership  of  the  product,  and  being  able  to  do  the  coding  at  OEC&S  helps  to 
eliminate  miscommunication.  The  coders  expressed  a  sense  of  being  able  to  ! 

influence  technology  for  training  development. 

Among  the  advantages  of  the  system  is  the  fact  that  it  is  possible  to  code 
anything  which  has  been  filmed,  although  they  felt  that  they  had  not  understood 
the  system  well  enough  to  have  sufficient  appropriate  video  on  hand. 

The  coders  also  like  the  immediate  feedback  provided  by  the  IVCRTS.  Other 
training  systems  have  to  be  implemented  before  course  developers  get  feedback. 

However,  the  computer  aspect  of  this  sytem  is  both  challenging  and  provides 
immediate  feedback.  One  stated  that  it  was  "like  playing  Atari." 

Despite  the  subjective  nature  of  the  coding  process,  the  coders  are  fairly 
confident  regarding  the  accuracy  of  their  coding.  Of  the  approximately  60 
codes,  they  only  seriously  disagreed  over  2  or  3  segments.  Five  to  ten  others 
were  resolved  by  viewing  and  discussing  the  segments.  They  reached  consensus 
on  all  of  the  segments  and  ran  test  people  through  the  system  for  further 
verification.  The  fact  that  the  system  focuses  on  specific  behaviors  and  skills 
assists  in  the  coding  process.  They  have  coded  all  15  competencies  but  feel 
limited  by  the  fact  that  their  coding  capability  was  limited  to  15. 
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Among  disadvantages  of  the  system  mentioned  by  the  coders  are  that  only  a 
small  number  of  people  can  use  the  system  at  a  time.  There  is  also  no 
motivation  for  trainees  to  select  the  correct  criteria  or  indicators  of  a 
competency  as  they  are  not  scored  on  this.  In  addition,  there  is  a  problem 
reproducing  the  videotapes  as  the  color  loses  resolution  and  the  time  code  is  not 
reproduced.  This  means  that  each  reproduction  has  to  be  manually  re-coded. 

In  general,  however,  the  coders  felt  that  the  system  has  a  lot  of  potential  and 
they  are  looking  forward  to  the  second  generation.  Such  improvements  as  touch 
sensitive  screens,  branching  programs,  and  learning  profiles  will  enhance  a 
system  that  they  are  already  quite  enthusiastic  about. 

Reliability 

There  were  no  real  reliability  problems  due  to  the  system  itself  insofar  as  the 
coders  were  concerned.  They  felt  that  any  problems  they  had  were  self-imposed. 
The  system  requires  the  same  safeguards  as  any  computerized  system,  e.g.,  a 
specific  location  away  from  a  window  and  electronic  safeguards.  At  the  time  of 
the  interviews,  one  of  the  light  oens  was  not  working,  but  this  was  considered  to 
be  a  minor  problem.  They  did  recommend  putting  protective  tape  around  any 
disks  that  are  not  to  be  written  on. 

The  trainers  disagreed  rather  frequently  and  sometimes  strongly,  with  the  coding 
of  the  competencies.  This  issue  will  be  discussed  in  more  detail  in  the  discussion 
of  the  training  system. 

In  summary,  the  inputting  and  coding  system  was  very  favorably  received  by  the 
coders.  While  complex,  they  found  it  easy  to  learn  to  use.  They  had  no  major 
problems  with  the  reliability  of  the  system.  Although  they  would  like  more 
flexibility  in  the  program,  they  were  quite  imoressed  with  the  system. 
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TRAINING  SYSTEM 


Effectiveness 


The  basic  question  to  be  answered  regarding  this  issue  is  whether  or  not  70%  of 
the  20  trainers  in  the  sample  learned  to  correctly  identify  70%  of  the  15 
competencies  as  coded  and  displayed  on  the  IVCRTS. 

The  performance  data  collected  via  the  IVCRTS  software  program  (Tables  7  and 
8)  do  not  directly  provide  a  clear  answer  to  the  basic  question.  That  is,  the 
quantitative  data  in  these  tables  are  insufficient  in  detail  and  unsuitable  in 
format  to  lead  directly  to  a  clear  conclusion  in  this  regard.  Nevertheless,  the 
researchers  interpreted  the  total  data  (i.e.,  the  IVCRTS  data,  questionnaire  data, 
and  interview  data)  to  support  a  general  statement  that  70%  of  the  sample  of 
trainers  did  not  sufficiently  demonstrate  an  ability  to  correctly  and  consistently 
identify  70%  of  the  15  competencies  coded  and  displayed  on  the  IVCRTS. 

As  an  absolute  minimum,  a  trainer  would  have  had  to  record  at  least  one  70% 
correct  session  in  "Name  It-Assessment,"  one  70%  correct  session  in  the  "Name 
It-Evaluation",  and  one  70%  correct  session  in  the  "Find  It-Planning"  exercises  to 
reach  criterion.  And  even  this  assumes  that  in  each  session  all  available 
competencies  were  displayed.  It  also  assumes  that  those  incorrectly  identified 
were  none  of  3  different  competencies  on  each  tape  that  are  not  available  on  the 
other  2  tapes  (total  of  9  competencies). 

Referring  to  Table  7,  it  is  noted  that  9  trainers  (or  75%)  recorded  2  successful 
(at  least  70%)  "Name  It"  sessions  and  l  successful  "Find  It"  session.  It  is  noted 
also  that  all  9  recorded  1  or  more  unsuccessful  (less  than  70%)  sessions  after  a 
minimum  number  of  successful  sessions  were  recorded.  Furthermore,  5  of  these 
9  trainers  recorded  an  unsuccessful  last-session  in  either  the  "Name  It"  or  "Find 
It"  exercise.  Altogether,  5  last-session  "Name  It"  sessions  and  six  last-session 
"Find  It"  sessions  were  recorded  at  below  the  70%  criterion.  Mean  scores 
computed  and  provided  in  Tables  7  and  8  also  support  the  conclusion  that  the 
established  criterion  was  not  reached.  For  example,  in  Table  7,  only  54%  of  the 


trainers  had  mean  session  scores  at  or  above  70  in  "Name  It,"  and  only  34%  had 
mean  session  scores  at  or  above  70  in  "Find  It." 

This  conclusion  is  not  meant  to  be  a  criticism  of  the  IVCRTS.  Although  the 
IVCRTS  performance  data  were  insufficient,  there  were  several  other  factors 
which  had  more  direct  influence  on  the  effectiveness  issue. 

Both  the  trainer  interviews  and  the  management  interviews  provided  information 
that  clearly  indicates  that  this  test  effort  was  not  well  received  by  the  trainers 
in  the  first  place.  They  were  not  willing  participants.  They  obviously  considered 
their  participation  an  added  burden  that  competed  with  other  work  requirements. 
They  were  eventually  ordered  to  participate  according  to  a  monitored  schedule, 
and  as  information  from  their  managers  show,  they  resented  such  pressure. 
Several  trainers  indicated  that  their  only  objective  during  their  training  sessions 
was  to  figure  out  how  to  "beat  the  machine"  in  order  to  reach  the  desired 
criterion.  Two  of  the  trainers  recorded  only  one  training  session  and  a  third 
never  recorded  a  score  of  70  or  more  during  two  sessions  of  "Name  It." 

The  trainers  frequently  and  sometimes  strongly  disagreed  with  the  coding  of  the 
competencies.  This  was  revealed  repeatedly  in  the  interviews.  Forty-two 
percent  of  the  trainers  interviewed  said  the  IVCRTS  did  not  adequately  assess 
their  level  of  competency  recognition.  Another  42%  answered  the  same  question 
with  a  qualified  "yes."  The  qualification  was  most  often  stated,  "yes,  if  we 
accept  the  assumption  that  the  competencies  were  coded  accurately  in  the  first 
place." 

When  asked  how  the  IVCRTS  system  could  be  improved,  47%  of  the  suggestions 
dealt  with  ways  to  improve  the  coding  process.  The  researchers  were  unable  to 
establish  any  pattern  or  draw  any  meaningful  conclusions  from  trainer  interview 
comments  concerning  specific  competency  presentations  (see  summary  at  Table 
12). 
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TABLE  12:  TRAINER  COMMENTS 
COMPETENCY  RECOGNITION  AND  DISPLAY 


Competency 

N=15 

Harder 
to  T rain 
Recognition 
N=17 

Best 

Displayed 

N=17 

Worst 

Displayed 

N=  14 

Establishes  Professional 

Rapport 

.29  (4) 

Seizes  Opportunities 

.24  (4) 

Results  Orientation 

.06  (1) 

.18(3) 

Aware  of  Self  Impact 

•  18  (3) 

.06  (1) 

Uses  Influence  Strategies 

.12  (2) 

.07(1) 

Seeks  Multiple  Perspectives 

.12  (2) 

Problem  Focused  Procedures 

.12  (2) 

Low  Fear  of  Rejection 

.12  (2) 

.12  (2) 

.07(1) 

Concern  for  Clarity 

.12(2) 

.06  (l) 

.07(1) 

Self  Confidence 

.06  (l) 

.12  (2) 

.14  (2) 

Perceptual  Objectivity 

.06(1) 

.07  (1) 

Identifies  Key  Themes 

.06  (1) 

.06(1) 

.07(1) 

Causal  Thinking 

.06  (1) 

.14  (2) 

Seeks  User  Input 

.06(1) 

Functional  Knowledge 

.06(1) 

.07(1) 

The  trainers  do  seem  to  think  that  a  larger  group  of  experienced  trainers/ 
consultants  (i.e.,  an  expert  panel)  should  be  involved  in  the  coding  process. 

From  the  Problem  Survey/Log  it  is  noted  that  57%  of  the  respondents  said  they 
were  not  satisfied  with  the  reason  provided  when  they  did  not  correctly  identify 
a  competency.  (This  response  differed  noticeably  between  one-time  users  (.63) 
and  multiple  users  (.43).) 

In  the  questionnaire  data,  38%  of  the  respondents  said  they  thought  the  IVCRTS 
was  effective  in  training  competency  recognition,  and  another  38%  said  the 
system  was  somewhat  effective. 

In  comparing  potential  methods  of  training  competency  recognition,  the  respon¬ 
dents  think  Structured  Experiences,  Role  Playing,  and  Case  Study  more  effective 
than  the  IVCRTS.  (These  are  primary  methods  currently  employed  in  the  OEC&S 
classrooms.)  IVCRTS  is  considered  more  effective  than  Written  Programmed 
Learning,  Self  Study  Tapes,  and  TV  Tapes. 

Another  general  criticism  expressed  by  the  trainers  concerns  their  perception 
that  competency  recognition  is  not  necessarily  relevant  to  their  work  in  the 
classroom.  Several  trainers  attempted  to  show  us  that  there  is  no  direct 
connection  between  the  competencies  and  classroom  learning  objectives.  When 
asked  in  what  way  their  recognition  skills  had  been  used  in  the  classroom  since 
their  IVCRTS  training,  58%  said  they  had  not  used  the  skills  while  25%  said  they 
were  more  sensitive  to  competency  behaviors  and  8%  said  they  now  use  more  of 
the  competency  language  in  the  classroom. 

Eighty-three  percent  of  the  trainers  interviewed  said  the  IVCRTS  training  had 
not  helped  them  do  their  job  better  or  easier. 

Twenty-nine  percent  of  these  trainers  said  they  thought  they  are  now  more 
aware  of  competency-type  behavior. 


Two  of  the  three  managers  interviewed  said  they  had  observed  no  change  in  the 
behavior  of  trainers  since  the  training.  The  other  manager  said  he  was  not  sure 
if  he  had  noticed  any  changes. 

Two  of  the  three  managers  said  they  had  no  evidence  that  the  trainers  were 
better  able  to  recognize  competencies;  the  other  thought  he  had  heard  a  little 
more  behavioral  terms  and  competency  language  from  his  trainers. 

Two  managers  said  the  IVCRTS  training  had  not  yet  contributed  to  training 
effectiveness  and  the  third  manager  said  he  could  not  answer  the  question. 

Efficiency 

There  are  no  firm  data  to  support  a  conclusive  finding  regarding  the  efficiency 
of  the  IVCRTS  as  it  was  employed  in  this  evaluation  effort.  It  is  not  known,  for 
example,  how  many  trials  or  how  much  time  was  required  to  train  a  trainer  to 
consistently  recognize  a  given  competency.  The  mean  score  computations 
provided  in  the  IVCRTS  data  tables  are  not  encouraging  in  this  regard.  There 
appears  to  be  no  strong  trend  in  improved  performance  across  sessions. 

Thirty  three  percent  of  the  trainers  interviewed  think  the  system  has  good 
potential  as  an  efficient  training  method  if  improved  upon;  25%  think  it 
compares  favorably  with  other  methods;  and  25%  think  it  is  the  best  method. 

The  system  was  readily  accessable.  Only  one  trainer  said  he  could  not  use  the 
system  when  he  wanted  to. 

Fifty  percent  of  the  trainers  said  the  Rundown  feature  was  not  useful;  17%  did 
not  remember  what  the  feature  was;  and  33%  said  the  feature  was  useful.  As 
the  coders  have  since  learned  and  reported,  this  feature  was  rarely  employed 
because,  as  now  programmed,  it  requires  a  single  competency  to  be  confused 
with  another  a  minimum  of  three  times.  This  apparently  did  not  occur  during  the 
sessions  recorded. 
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User  Acceptance 


From  the  evaluation  questionnaire,  it  is  noted  that  user  interest  in  the  system 
ranged  from  "very  bored"  to  "very  interested,"  with  the  mean  response  being 
midway  between  "neutral"  and  "interested."  Thirty-eight  percent  of  the  16 
respondents  indicated  they  would  definitely  like  to  see  IVCRTS-type  technology 
used  for  additional  training  at  OEC&S.  Forty-four  percent  would  recommmend 
the  IVCRTS  very  highly  to  a  friend  looking  for  a  way  to  train  recognition  skills. 
Twenty-five  percent  would  recommend  it  with  reservations. 

During  the  trainer  interviews,  26  "best  features"  of  the  system  were  cited, 
including:  interactive;  interesting/novel;  user  friendly;  competency  presen¬ 

tations;  convenience;  instant  feedback;  quality  of  the  film;  and  concentration  on 
a  narrow  set  of  learning  objectives. 

Of  the  29  suggested  improvements,  17  (or  59%)  dealt  with  ways  to  in. prove  the 
competency  presentations.  These  included:  better,  more  realistic  coding  (from 
8  trainers),  separate  scenarios  for  "Name  It"  and  "Find  It";  brief,  concise 
competency  definitions;  limit  number  of  competencies  considered  to  12;  elim¬ 
inate  need  to  identify  the  competencies  in  a  prescribed  order,  provide  study 
material  before  training;  and  develop  scenarios  for  specific  competencies. 

When  asked  for  an  overall  reaction  to  the  IVCRTS,  7  (or  56%)  gave  generally 
negative  comments,  while  5  (or  42%)  gave  generally  positive  comments. 

Reliability 

The  IVCRTS  is  a  highly  sophisticated  and  complex  system.  The  coders  are  still 
learning  how  to  interact  with  it  and  how  it  may  be  used  to  best  advantage. 

Not  all  of  the  available  IVCRTS  features  were  well  used  by  the  trainers  who 
tested  the  system.  As  an  example,  of  the  22  users  who  completed 
Problem/Survey  Logs,  only  32%  indicated  they  had  used  the  Introduction  feature. 
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.  the  previous  section,  this  feature  is  the  primary  means  of 
•  ••  ■»'  *  i' «  the  uer  to  the  competency  model,  definitions,  and  behavioral 

■'til  *?ovv 

The  r  jt<  r  *as  me  least  used  feature  of  the  system.  Only  7%  of  first  time  users 
engaged  the  Tutor,  while  14%  of  multiple  users  did  so. 

As  would  be  expected,  the  "Name  It"  exercise  was  played  by  most  first-time 
users  (93%),  while  the  "Find  It"  exercise  was  played  by  all  multiple  session  users. 

The  Reliability  Score  feature  was  used  heavily  by  first-time  users  (87%)  and 
much  less  by  multiple  session  users.  This  may  indicate  that  those  who  continued 
to  use  the  system  for  three  or  more  sessions  were  less  concerned  with  reliability 
scores. 

Several  problems  were  encountered  by  the  trainers.  Fifty-three  percent 
indicated  they  experienced  occasions  when  they  did  not  know  what  to  do  next. 
This  became  less  of  a  problem  for  those  who  continued  to  work  with  the  system 
during  multiple  sessions. 

A  significant  and  continuing  problem  for  most  users  involved  the  use  of  the  light 
pen.  Seventy-six  percent  of  the  respondents  indicated  they  touched  the  panel 
and  nothing  happened.  According  to  the  coders,  the  system  will  not  respond 
unless  the  pen  touches  the  panel  slightly  below  a  desired  item  and  on  a  bright 
portion  of  the  screen. 

When  asked  in  the  interviews  about  consistent  problems  with  the  system,  58%  of 
the  trainers  said  they  had  difficulty  with  the  light  pen;  25%  said  they  had 
problems  getting  the  equipment  to  work;  and  17%  said  they  had  trouble  getting 
their  profile  scores. 

Generally,  the  Rundown  feature  was  not  found  to  be  useful,  primarily  for  reasons 
discussed  earlier. 
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TRAINING  MANAGEMENT  SYSTEM 


Cost  Benefit 

The  training  management  mechanism  appears  to  have  succeeded  well  in  record¬ 
ing  accurately  the  performance  computations  it  is  currently  programmed  to 
make.  The  system  now  computes  for  each  trainer  the  percentage  of  successful 
recognitions  for  each  competency  and  displays  the  computation  in  the  form  of  a 
histogram.  The  system  also  computes  aggregate  scores  of  the  type  displayed  in 
Tables  7  and  8. 

As  has  been  discussed,  the  system  does  not  provide  information  concerning  which 
competencies  are  drilled  by  session,  with  what  frequency  they  are  presented,  or 
in  what  sequence  they  are  presented.  Nor  is  there  trial  data  that  would  allow 
one  to  follow  the  progression  of  trial  performance  to  the  desired  criterion.  Such 
data  are  required  to  conduct  a  cost  benefit  analysis  of  the  type  intended  when 
the  evaluation  plan  was  prepared. 

Effectiveness 

Most  data  items  which  address  this  issue  are  taken  from  the  Problem  Survey/ 
Logs  and  the  Evaluation  Questionnaire. 

None  of  the  trainers  found  the  terminology  contained  in  the  system  confusing, 
nor  was  the  system  availability  a  problem. 

In  responding  to  the  question,  "How  adequate  is  the  score  display  regarding  your 
performance?",  the  trainers  rated  the  display  "adequate"  to  "good."  And,  as  has 
already  been  mentioned,  42%  of  the  trainers  thought  the  system  did  not 
adequately  assess  their  level  of  competency  recognition,  while  42%  qualified 
their  answer  that  the  system  did  adequately  assess  their  competency  level. 


SECTION  V:  SUMMARY  AND  RECOMMENDATIONS 


SUMMARY 

This  analysis  of  available  data  led  to  the  following  summary  conclusions: 

1.  The  IVCRTS  is  a  highly  complex  and  sophisticated  training  apparatus 
that  is  potentially  very  useful  in  the  OEC&S  environment,  especially 
if  combined  with  other  training  methods. 

2.  The  IVCRTS  Inputting  and  Coding  System  requires  several  weeks  of 
study  and  hands-on  experience  to  enable  coders  to  use  it  to  its  full 
potential. 

3.  The  IVCRTS  Training  System,  as  it  was  employed  in  this  evaluation 
effort,  did  not  bring  about  the  training  results  desired. 

4.  Factors  which  contributed  most  to  the  training  performance  obtained 
were: 

•  Limitations  inherent  in  the  videotapes  used; 

•  The  nature  of  the  skills  being  trained; 

•  The  subjectivity  involved  in  the  coding  process; 

•  Absence  of  strong  motivation  of  the  sample  of  trainers; 

•  Conditions  under  which  the  evaluation  was  conducted; 
and, 

•  Absence  of  needed  data. 

5.  The  IVCRTS  Training  Management  System  does  not  record  data  in 
sufficient  detail  or  appropriate  format  to  adequately  assess  training 
gains. 
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RECOMMENDATIONS 


Continue  Evaluation 


It  is  recommended  that  OEC&S  continue  to  evaluate  the  effectiveness  and 
efficiency  of  the  IVCRTS  in  the  OECicS  setting.  The  majority  of  the  people  who 
contributed  to  the  evaluation  effort  appear  to  think  that  there  is  a  place  for  the 
IVCRTS  within  OEC&S.  Most  trainers,  it  seems,  would  be  interested  in  using  the 
IVCRTS  in  combination  with  one  or  more  of  the  training  methods  now  employed 
in  the  classroom.  The  training  managers  are  particularly  optimistic  about  the 
future  use  of  the  IVCRTS,  particularly  in  the  areas  of  standardization  of 
instruction  and  grading.  The  training  developers  view  the  IVCRTS  as  an 
excellent  resource  and  are  looking  forward  to  expected  improvements  in  the 
system. 

Use  Panel  of  Experts  to  Code 

The  apparent  lack  of  success  of  the  IVCRTS  in  this  case  may  have  resulted  more 
from  the  nature  of  the  skills  being  trained  (i.e.,  the  McBer  competencies)  than 
from  the  system  itself.  The  consultant-client  interaction  is  a  highly  complex  one 
that  requires  the  successful  consultant  to  demonstrate  a  large  number  of 
competencies,  often  simultaneously.  Any  selected  segment  of  sequential 
behaviors  is  so  rich  with  possible  combinations  that  a  forced  choice  of  some 
single  and  minute  part  is  artificial  at  best.  Add  to  this  the  highly  subjective 
nature  of  the  coding  process,  and  the  chances  of  reaching  a  meaningful  level  of 
agreement  diminishes  significantly. 

Nevertheless,  agreement  could  probably  be  improved  upon  in  this  case  if  more 
experienced  people  were  involved  in  the  coding  process.  It  is  recommended, 
therefore,  that  a  larger  panel  of  subject  matter  experts  with  demonstrated 
success  as  both  instructors  and  consultants  be  involved  in  the  competency  coding 
process. 
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Learning  objectives  for  each  block  of  instruction  apparently  do  not  incorporate 
specific  competencies  or  competency  based  behaviors.  If,  in  fact,  an  objective 
of  the  Program  of  Instruction  is  to  train  the  essential  competencies  identified  in 
the  McBer  study,  then  it  seems  appropriate  that  learning  objectives  incorporate 
specific  competencies  as  appropriate  to  each  objective. 

Evaluate  IVCRTS  Through  Student  Use 

Assuming  that  the  competencies  support  classroom  learning  objectives  and  that 
an  expert  panel  can  agree  to  realistic  coding  procedures,  it  seems  the  IVCRTS 
has  more  potential  for  use  as  a  training  mechanism  for  students  in  the 
consultant's  course.  In  addition  to  assisting  in  the  standardization  of  instruction 
and  grading,  the  IVCRTS  could  be  used  to  compliment  the  existing  methods  of 
instruction.  Furthermore,  since  the  students'  activities  are  controlled  and 
monitored  closely,  it  is  likely  that  the  effectiveness  and  efficiency  of  the 
IVCRTS  could  be  more  accurately  assessed  with  such  a  group.  The  students  are 
also  more  likely  to  be  motivated  to  use  the  system  to  its  full  potential. 

Modify  the  Individual  Training  Management  System 

With  more  control  over  the  users  of  the  IVCRTS  (i.e.,  students  in  the  course)  and 
perhaps  with  additional  program  software,  it  should  be  possible  to  obtain  more 
and  better  data  to  assist  in  evaluating  the  effectiveness  and  efficiency  of  the 
system.  As  a  minimum,  there  needs  to  be  a  record  of  which  competencies  are 
presented  in  each  session,  in  what  order,  and  with  what  frequency.  Additionally, 
the  amount  of  time  devoted  to  each  session  should  also  be  recorded.  These  data 
should  be  available  for  each  individual  and  for  groups  of  individuals. 


Also,  for  maximum  effect,  students  should  be  required  to  demonstrate  ability  on 
the  system  individually  rather  than  in  pairs. 


Locate/Develop  Other  Videotapes 


Finally,  it  is  recommended  that  additional  videotapes  be  located  and/or  devel¬ 
oped  to  allow  for  both  the  coding  of  additional  competencies  and  the  coding  of 
the  same  competencies  in  different  settings  or  contexts.  If  possible,  the 
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APPENDIX  A 

IVCRTS  DATA 


Percent  Competency  Presentation  Correct  by  Session  and 
Trainer(s)  for  Name-It. 


Percent  Competency  Presentation  Correct  by  Session  and 
Trainer(s)  for  Find-It. 


Percent  Correct  Recognition  for  Each  Competency  in  Name-It. 
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Percent  Correct  Recognition  for  Each  Competency  in  Find-It. 


Trainer 


Session 


Mean 


57 

100 

60 

60 

54 

100 

100 

- 

76 

71 

60 

90 

- 

- 

- 

- 
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74 
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- 

67 

43 

100 

50 

100 

- 

- 

- 

73 

43 

86 

33 

60 

- 

- 

- 

- 

56 

57 

100 

50 

80 

- 

- 

- 

72 

29 

40 

- 

- 

- 

- 

- 

- 

35 

83 

70 

- 

- 

- 

- 

- 

- 

77 

86 

70 

- 

- 
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- 

- 

- 

78 

79 

100 

100 
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100 

100 

89 

100 
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46 

80 

64 

18 

43 

- 

55 

62 

75 
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76 

64 

70 
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TABLE  2 

PERCENT  COMPETENCY  PRESENTATION  CORRECT  BY 
SESSION  AND  TRAINER(S)  FOR  FIND-IT 


T  rainer 

Session 

Mean 

1 

2 

3 

4 

5 

6 

7 

1 

50 

91 

18 

o 

35 

47 

- 

40 

2 

67 

100 

80 

30 

47 

58 

100 

69 

3 

44 

65 

81 

- 

- 

- 

- 

63 

4 

50 

100 

58 

71 

- 

- 

- 

70 

5 

100 

33 

50 

44 

70 

- 

- 

59 

6 

71 

70 

33 

- 

- 

- 

- 

58 

7 

86 

100 

75 

- 

- 

- 

87 

8 

50 

100 

64 

80 

43 

40 

- 

63 

9 

86 

100 

38 

93 

- 

- 

- 

79 

10 

88 

77 

22 

- 

- 

62 

11 

82 

- 

- 

- 

82 

12 

50 

- 

- 

- 

- 

- 

- 

50 

13 

43 

- 

- 

- 

- 

- 

- 

43 

Mean 

67 

84 

52 

53 

49 

48 

100 

%j£70 

31 
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TABLE  3 

PERCENT  CORRECT  RECOGNITION  FOR  EACH  COMPETENCY 

IN  NAME-IT 


Competency 

Total 

Number 

Presentations 

Percent 

Correct 

Recognitions 

Functional  Knowledge 

65 

63 

Self  Confidence 

37 

70 

Low  Fear  of  Rejection 

64 

77 

Establishes  Professional  Rapport 

34 

89 

Concern  for  Clarity 

112 

72 

Seeks  User  Input 

12 

25 

Uses  Influence  Strategies 

8 

25 

Communicates  Clearly 

33 

61 

Aware  of  Self  Impact 

33 

70 

Seeks  Multiple  Perspectives 

40 

75 

Causal  Thinking 

30 

77 

Identifies  Key  Themes 

78 

83 

Problem  Focused  Procedures 

13 

54 

Seizes  Opportunities 

32 

100 

Results  Orientation 

38 

69 

Mean  =  67.30 
SD  =  20.50 
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TABLE  4 

PERCENT  CORRECT  RECOGNITION  FOR  EACH  COMPETENCY 

IN  FIND-IT 


Competency 

Total 

Number 

Presentations 

Percent 

Correct 

Recognitions 

Functional  Knowledge 

44 

66 

Self  Confidence 

56 

70 

Low  Fear  of  Rejection 

43 

79 

Establishes  Professional  Rapport 

23 

83 

Concern  for  Clarity 

175 

58 

Seeks  User  Input 

70 

52 

Uses  Influence  Strategies 

34 

35 

Communicates  Clearly 

27 

52 

Aware  of  Self  Impact 

24 

67 

Seeks  Multiple  Perspectives 

61 

59 

Causal  Thinking 

21 

95 

Identifies  Key  Themes 

117 

69 

Problem  Focused  Procedures 

85 

48 

Seizes  Opportunities 

21 

95 

Results  Orientation 

71 

45 

Mean  =  64.90 
SD  =  17.68 

COURSE  DEVELOPERS  INTERVIEW  COMMENTS 


(Two  Course  Developers  Were  Interviewed) 


1.  Whit  advantages/disadvantages  do  you  see  for  course  developers  using 
interactive  video  training  systems? 

Advantages: 

o  Other  systems  require  inputting  and  coding  to  be  done  at 
corporation  while  IVCRTS  can  be  done  on-site  -it  eliminates 
miscommunication. 

o  New  technology  for  training  development. 

o  Multi-media  -  anything  on  film  can  be  used. 

o  Other  systems  have  to  be  implemented  before  course  developers 
get  feedback,  IVCRTS  allows  immediate  feedback. 

o  Focuses  on  specific  behaviors,  skills. 

Disadvantages: 

o  Cannot  code  for  8  hours  -  need  to  use  all  senses  and  be  alert  -it's 
too  intense. 

o  No  motivation  for  student  to  select  correct  criteria  -  is  not  scored. 

o  Very  complex  coding  system:  9  functions,  23  tasks,  26  operations  - 
need  to  understand  operations  to  choose  function  tool.  Need 
formal  training  to  use:  5  days  -  2  weeks. 

But  a  flexible  system  requires  complexity. 

o  Can  only  use  with  small  number  of  people  at  a  time. 

o  Tied  into  set  software  -expensive  to  change. 

o  Easy  to  confuse  -  "concern  for  clarity"  and  "seeks  user  input." 


2.  What,  if  any,  problems  did  you  encounter  in  using  the  IVCRTS  inputting  and 
coding  system? 

o  2  coders  is  optimal  -  3  would  be  too  many. 

o  There  is  a  tendency  for  2  coders  to  communicate,  but  if  one  was  at 

the  keyboard,  the  other  may  not  pay  attention. 
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COURSE  DEVELOPERS  INTERVIEW  COMMENTS 


o  The  subjective  nature  of  the  coding  can  lead  to  disagreement  on 
coding  a  segment  -  this  was  sometimes  solved  by  "who  yelled 
loudest." 

o  Have  reached  consensus  on  all  segments  -  coding  was  corrected  by 
running  test  people  through  system. 

o  Mistakes  occurred  after  working  on  system  too  long. 

o  One  light  pen  currently  not  working. 


Did  you  find  the  instructions  for  using  the  inputting  and  coding  system  clear 
and  unambiguous? 

o  Instructions  were  complete. 

o  Instructions  were  slim. 

o  There  was  no  information  overload. 

o  Very  user  friendly  -  it  asks  prior  to  erasing  input. 


Is  there  anything  you  would  like  to  see  changed  or  added  to  improve  the 
inputting  and  coding  system? 

Each  time  a  tape  is  copied,  it  loses  resolution.  It  would  be  better  to 
have  exact  time-coding;  now,  we  have  to  manually  recode. 

Need  to  change  to  touch-sensitive  screen. 

Problem: 


o  Only  16  competencies  on  system. 

o  Did  not  understand  system  well  e  >ugh  to  have  enough  appropriate 
video  on  hand. 

Main  advantage: 

o  Ability  to  code  any  video. 

Improvement  wanted: 

o  Tutor  is  only  keyed  by  "Find  It"  -  would  like  to  be  able  to  access  it 
by  "Name  It." 

o  Like  to  add  coder's  ability  to  add  or  delete  competencies. 
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Change: 


o  Ability  to  add  supporting  narrative  material  -  don't  produce  fully 
capable  course. 

Anxious  for  second  generation's  touch  screen,  learning  profile, 
branching  program. 


Did  you  ever  want  to  exit  the  program  and  not  be  able  to  easily? 

Problem  exiting  program: 

o  If  competencies  are  not  selected,  will  go  through  entire  program  - 
plan  to  change  it  to  requiring  selection  of  a  minimum  of  3 
competencies. 

o  If  a  segment  has  been  chosen,  it  has  to  be  coded  to  exit  the 
program  -  then,  if  necessary,  can  be  backed  and  erase. 

o  Some  tasks  will  not  allow  you  to  exit  even  if  you  made  the  wrong 
choice. 


Did  you  have  any  problems  modifying  the  windows  of  a  segment?  Why? 

o  Initial  problems  in  modifying  segment  windows  -  but  improved  with 
learning. 

o  At  first,  had  problem  modifying  the  windows  of  a  segment.  Later, 
there  was  no  problem. 

o  There  is  a  learning  curve  and  can  now  make  very  minute  changes. 


Were  you  able  to  locate  specific  segments  easily?  Why  or  why  not? 
o  Easy  to  locate  specific  segments. 

o  Very  easy  to  locate  specific  segments  -  can  view  and  stop  video 
codes  and  identifies  frame  number.  If  you  change  your  mind,  the 
segment  can  be  found  with  the  frame  number. 


COURSE  DEVELOPERS  INTERVIEW  COMMENTS 


8.  Did  you  ever  use  a  command  and  find  that  what  you  intended  to  happen  did 
not  occur?  How  frequently? 

o  If  you  have  disks  that  you  don't  want  to  be  written  on,  put 
protective  tape  around. 

o  Did  not  have  a  problem  using  a  command  and  finding  that  what  was 
intended  did  not  occur. 


9.  Did  you  encounter  any  difficulties  in  re-coding  competencies?  What? 
o  No  difficulties  in  recoding  competencies  -  flexible  system, 
o  No  difficulties  in  recoding  competencies. 


10.  Were  there  any  problems  obtaining  consensus  as  to  which  competency  was 
being  displayed?  If  so,  how  was  it  resolved? 

o  98%  agreement  between  coders.  1-2  segments  emphatic  disagree¬ 
ment.  5-10  talked  over. 

o  View  and  discuss  to  resolve  60  codes,  3  segments  disagreed. 


11.  Were  some  competencies  easier  to  identify  then  others?  Which  ones  were 
clear?  Which  ones  did  you  have  trouble  with? 

o  Certain  competencies  frequently  coded: 

Self  confidence 
Identify  key  themes 
Results  orientation 

o  Communicates  clearly  -  constantly  there,  not  coded  frequently. 

o  Tendency  to  mix  some  competencies: 

Concern  for  clarity  and  results  orientation. 

Self  confidence/low  fear  of  rejection. 

Concern  for  clarity/seeks  user  input. 

o  Difficult  to  identify: 

Aware  of  self  impact 
Perceptual  objectivity 
-  Causal  thinking 

Seizes  opportunities 
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o  Clearly  identified: 

Functional  knowledge 
Oral  presentation  skills 
Identify  key  themes 

o  Seeks  user  input  and  concern  for  clarity  easy  but  difficult  to 
capture  criteria. 

o  Certain  competencies  clearly  identified: 

Seeks  multiple  perspectives 
Seizes  opportunities 
Establishes  professional  rapport 
Functional  knowledge 

o  Have  coded  all  16  competencies. 

12.  How  confident  do  you  feel  about  the  accuracy  of  the  coding?  Why? 

o  Feel  80%  confident  of  accuracy  of  coding 
o  Pretty  confident  regarding  accuracy  of  coding 

13.  In  general,  did  you  find  the  inputting  and  coding  system  easy  to  use?  Why? 

o  Inputting  system  allows  freedom  to  do  coding  and  make 
corrections. 

o  Need  to  have  backup  manual  data  on  initial  coding, 
o  Takes  learning. 

o  Need  to  read  instructions  carefully. 

o  In  general,  found  inputting  and  coding  system  easy  to  use. 

14.  What  is  your  overall  reaction  to  the  inputting  and  coding  system?  Be 
explicit  and  objective. 

o  System  has  lots  of  potential 

o  Like  to  see  coders  trained  first. 

o  People  are  attracted  to  the  computer  aspect:  it  is  challenging  and 
provides  immediate  feedback  -  "like  playing  Atari." 


COURSE  DEVELOPERS  INTERVIEW  COMMENTS 


o  Impressed  with  inputting  and  coding  system. 

o  Had  a  sense  of  being  able  to  influence  training  technology,  not  vice 
versa. 


15.  Did  you  have  any  problems  with  the  hardware,  i.e.,  downtime,  maintain¬ 
ability,  etc.? 

o  No  reliability  problems  created  by  system, 
o  Problems  were  self-imposed. 

o  System  requires  a  specific  location,  electronic  safeguard,  and  must 
be  away  from  window  -  same  safeguards  as  any  computerized 
system. 


TRAINER  INTERVIEW  COMMENTS 


(12  trainers  were  interviewed:  9  (.75)  military  and  3  (.25)  civilian.  Where 
item  was  mentioned  by  more  than  one  person,  the  frequency  appears 
parenthesis.) 

1.  What  do  you  consider  to  be  the  best  features  of  the  IVCRTS?  (26  Items) 

o  Interactive/Instant  feedback.  (5)  .20 
o  Interesting/novel.  (3)  .12 
o  User  friendly.  (3)  .12 
o  Competency  representation.  (3)  .12 
o  Convenience.  (2)  .08 

o  Quality  of  film.  (2)  .08 

o  Concentration  on  narrow  learning  objectives.  (2)  .08 
o  Quality  of  graphics, 

o  Realistic  consulting  activity, 
o  Team  training. 

o  There  were  no  useful  features.  (2)  .08 

2.  How  could  the  IVCRTS  be  improved?  (29  Items) 

Competencies:  (17  Items/.59) 
c  Better,  more  realistic  coding.  (8)  .47 
o  Separate  scenarios  for  "Name  It"  and  "Find  It."  (3)  .07 
o  Brief,  concise  definitions.  (2)  .07 
o  Limit  number  considered  to  twelve, 
o  Eliminate  need  to  identify  in  prescribed  order, 
o  Study  material  before  training, 

o  Develop  scenarios  for  specific  competencies. 


TRAINER  INTERVIEW  COMMENTS 


Hardware/Software:  (10  Items/.34) 
o  Synchronize  disks  and  tapes.  (3)  .30 
o  More  responsive  light  pen.  (3)  .30 
o  Less  equipment  downtime.  (2)  .20 
o  Provide  a  way  to  correct  accidental  touch  of  screen, 
o  More  detailed  operating  instructions. 

Videotape:  (2  Items/.07) 
o  More  realistic/natural  action, 

o  Better  quality  of  film. 


Which  of  your  skills  do  you  believe  have  been  improved  by  the  IVCRTS?  (14 


o  Awareness  of  competency-type  behavior.  (4)  .36 
o  Using  competency  language, 

o  Using  the  hardware.  (3)  .21 

o  Taking  tests.  (2)  .14 
o  Memory, 
o  None.  (3)  .21 


In  what  ways  have  you  used  the  skills  trained  by  the  IVCRTS  in  the 
classroom?  (12  Items) 

o  Have  not  used  skills.  (7)  .58 

o  More  sensitive  to  competency  behaviors.  (3)  .25 

o  Communicate  in  competency  language. 

o  Haven't  been  in  classroom. 
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Do  you  think  the  training  provided  by  the  IVCRTS  has  helped  you  do  your  job 
better  or  more  easily?  Whyt  ( 1 2  Items) 

o  No.  (10)  .83 

o  Not  sure. 

o  Somewhat,  as  a  catalyst  for  discussion. 

Do  you  think  that  it  is  less  easy  to  train  recognition  of  certain  competencies 
on  the  IVCRTS  than  others?  Which  ones?  (17  Items) 

-  Yes  (7):  .58 

o  Aware  of  self  impact.  (3)  .18 

o  Low  fear  of  rejection.  (2)  .12 

o  Concern  for  clarity.  (2)  .12 

o  Uses  influence  strategies.  (2)  .12 

o  Seeks  multiple  perspectives.  (2)  .12 

o  Problem  focused  procedures.  (2)  .12 

o  Self  confidence, 

o  Perceptual  objectivity, 

o  Identifies  key  themes 

o  Results  orientation. 

No.  (4)  .33 

There  was  no  training  involved.  (2)  .17 

Which  competencies  do  you  think  were  best  displayed?  Worst? 

Best:  (17  Items) 
o  Seizes  opportunities.  (4)  .24 

o  Results  orientation.  (3)  .18 
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o  Self  confidence.  (2)  .12 

o  Low  fear  of  rejection.  (2)  .12 

o  Concern  for  clarity, 

o  Aware  of  self  impact, 

o  Causal  thinking, 

o  Identifies  key  themes, 

o  Seeks  user  imput. 

o  Functional  knowledge. 

Worst:  (15  Items) 

o  Establishes  professional  rapport.  (4)  .27 
o  Self  confidence.  (2)*  .13 

o  Causal  thinking.  (2)  .13 

o  Concern  for  clarity, 

o  Identifies  key  themes, 

o  Perceptual  objectivity.* 

o  Uses  influence  strategies.* 
o  Low  fear  of  rejection.* 

o  Functional  knowledge.* 

o  None  were  done  well. 

Can't  answer/don't  remember.  (5)  .42 

These  were  difficult  because  they  can  be  perceived  as  involving  multiple 

competencies. 
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8.  Did  you  find  the  "Rundown"  feature  useful? 

-  No  (6):  .50 

.  o  It  was  of  no  use  even  when  we  thought  we  had  problems.  (4) 
o  It  was  not  useful.  (2) 

-  Yes  (4):  .33 
o  It  was  useful.  (3) 
o  It  was  useful  if  I  ageed  with  coding. 

Don't  remember  using  it.  (2)  .17 

9.  Did  you  have  any  consistent  problems  with  the  IVCRTS? 

o  Light  pen  performance.  (7)  .58 
o  Equipment  wouldn't  function.  (3)  .25 
o  Couldn't  get  profile  scores.  (2)  .17 

o  Hard  to  concentrate  on  everything,  i.e.,  video/audio,  narration,  and 
competencies.  (2)  .17 

o  "Rundown"  wouldn't  work. 

o  Understanding  definitions. 

o  No  problems. 

10.  Do  you  think  the  IVCRTS  adequately  assessed  your  level  of  competency 
recognition? 

o  No.  (5)  .42 

o  Yes,  if  one  accepts  the  definitions  and  accuracy  of  coding.  (5)  .42 
o  Only  for  those  which  were  displayed  well. 


o  Not  sure. 
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How  does  the  IVCRTS  compare  to  other  potential  methods  of  training 
competency  recognitions  in  terms  of  efficient  use  of  your  time? 

o  Has  good  potential  if  improved  on.  (4)  .33 

o  It  compares  favorably  with  other  methods.  (3)  .25 

o  It  is  the  best  method.  (3)  .25 

o  It  is  least  desirable  method. 

o  Don't  know. 

What  is  your  overall  reaction  to  the  IVCRTS? 

Negative;  (7)  .58 

o  It  is  unsatisfactory  in  terms  of  time  invested  and  resulting  payoff. 

o  It  was  interesting,  but  was  basically  a  guessing  game. 

o  Extreme  disappointment. 

o  It  was  fun,  but  didn't  learn  anything. 

o  It  takes  a  fairly  easy  task  and  makes  it  more  difficult. 

o  Negative;  the  way  we  were  forced  to  use  it  within  a  prescribed 
time  period  did  not  help  any. 

o  It  is  spending  a  lot  of  money  on  something  that  will  look  good,  but 
won’t  be  very  useful  in  the  classroom. 

Positive:  (5)  .42 

o  Enormous  potential;  software  preparation  and  validation  will  make 
all  the  difference. 

o  A  good  means  of  instruction  if  combined  with  role  paying  and 
audiovisual  feedback  to  the  role  player. 

o  It  is  a  step  in  the  right  direction;  needs  fine-tuning. 

o  Enthusiastic;  it  has  a  lot  of  potential. 

o  It  has  potential  but  the  programming  needs  to  be  reviewed. 
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What  Army-wide  applications  do  you  see  for  this  type  of  training  system? 

o  Not  sure.  (2) 

o  With  hard-data  skill  training,  i.e.,  where  there  is  a  clear  right  or 
wrong  answer,  e.g.,  math,  spelling,  etc.  (2) 

o  If  properly  developed  and  functioning,  it  could  be  used  to  select 
people  for  jobs. 

o  SQT  training. 

o  Training  in  the  area  of  interpersonal  skills,  e.g.,  leadership,  super¬ 
visory,  counseling,  communications. 

o  Maintenance  training,  weapons  training,  and  mechanical  pro¬ 
cedures. 

o  Introducing  new  equipment. 

o  Situational  leadership  training  to  NCOs  and  officers. 

o  Unlimited  potential  for  all  types  of  training. 

o  I'm  afraid  it  will  be  applied  Army-wide  without  being  tested. 


Can  you  recommend  any  other  videotapes  which  could  be  coded  for  use  at 
OEC&S? 


o  No.  (5)  M2 

o  No,  tapes  should  be  designed  and  developed  especially  for  this 
purpose.  (3) 

o  Yes,  the  movie  "12  O'Clock  High"  has  been  used  in  other  places  to 
teach  leadership.  (2) 

o  Yes,  the  Situational  Leadership  tapes  TDI  (Training  Development 
Institute)  developed  for  the  Drill  Sergeants  School. 
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15.  Any  other  comments? 

o  The  idea  is  good,  but  it  needs  more  work. 

o  They  need  to  be  more  specific  in  their  definitions  of  the  compe¬ 

tencies. 

o  The  concept  is  good  and  the  system  could  be  useful  if  the  coding 
problems  and  machine  operating  problems  are  resolved. 

o  I'm  afraid  it  will  be  implemented  before  it  is  refined.  They  need  to 
throw  the  competencies  out  and  start  all  over. 

o  I  have  serious  reservations  about  buying  it  as  it  is.  The  evaluation 
should  be  done  over  a  more  reasonable  time  period  and  users  should 
understand  what  the  outcomes  are  and  it  needs  to  be  phased  into 
what's  going  on  in  the  classroom. 

o  Something  went  wrong  with  the  coding  -  we  were  laughing  in  the 
aisles  the  first  time  through.  There  is  little  relationship  between 
the  competencies  and  the  performance  objectives  for  each  segment 
of  the  course  we  teach. 

o  My  fear  is  that  someone  has  already  decided  that  we  need  this  and 
we  don't.  It  was  simply  a  measure  of  my  ability  to  play  the  game, 
i.e.,  associate  competency  language  with  behavior  on  the  tape.  It 
just  doesn't  have  any  applicability  to  what  we're  doing  in  the 
classroom. 

o  I'd  like  to  see  the  money  spent  on  improved  procedures  for 
selecting  our  students. 

1 6.  How  long  have  you  been  a  trainer?  Consultant? 

(mean=1.9,  var.  =  .63,  SD=.79)  (mean=3.4,  var.  =  l .5,  SD=1.2) 


Trainer 

1)  0-1  yr  (4) 

2)  1-2  yrs  (5) 

3)  2-3  yrs  (1) 

4)  over  8  yrs  (2) 


Consultant 

1)  0(1) 

2)  lyr(l) 

3)  1-2  yrs (5) 

4)  2-4  yrs  (2) 

5)  over  5  yrs  (3) 
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(Director  of  Training,  Deputy  Director  of  Training,  and  Division  Chief) 

What  changes  have  you  observed  in  the  trainers  since  the  IVCRTS  was 
implemented  which  you  attribute  to  the  IVCRTS? 

o  Probably  none. 

o  None. 

o  Not  sure  -  some  may  be  taking  more  of  a  behavior  focus  on 
training. 


Did  you  observe  any  problems  with  the  implementation  of  the  IVCRTS?  Did 
you  observe  any  resistance  on  the  part  of  the  trainers  to  the  IVCRTS? 

Implementations 

o  The  evaluation  plan  did  not  take  into  account  the  training  load  on 
our  trainers.  There  was  no  cooperative  effort  between  TD  and  DT 
on  this. 

o  It  was  an  additional  requirement  for  our  trainers. 

o  There  was  no  problem. 

o  I  hear  there  was  a  problem  with  the  use  of  old  film  with  the  new 
technology. 

Resistance; 

o  Our  trainers  didn't  like  having  to  devote  so  much  time  to  the  effort 
and  they  don't  seem  to  like  working  with  the  hardware  used  on  the 
system. 

o  Yes,  mainly  due  to  the  total  time  demands  on  the  trainers.  We  had 
to  direct  them  to  do  it. 

o  Yes.  They  just  wouldn't  go  over  there  and  do  it.  This  was  partly 
because  of  competing  demands  on  their  time  and  partly  because 
they  just  weren't  motivated. 

o  We  finally  had  to  put  out  a  schedule  for  them  to  follow,  and  they 
resented  that. 
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Do  you  think  that  the  trainers  are  better  able  to  recognize  competencies  as 
a  result  of  the  IVCRTS?  Why  or  why  not? 

o  I  do  not  have  any  facts  on  which  to  base  a  statement  that  they 
have.  I  just  assume  they  are. 

o  They  all  saw  the  same  thing,  so  they  should  have  a  common 
language. 

o  I'm  seeing  a  little  more  use  of  behavioral  terms  and  competency 
langauge. 

o  I  have  seen  no  demonstration  of  improvement.  I  would  think  that 
they  are  better  simply  because  they  spent  so  much  time  doing  it. 

What  do  you  consider  to  be  the  best  features  of  the  IVCRTS? 

o  Providing  a  common  base  of  competencies  for  all  trainers  to  work 
from. 

o  This  will  help  to  standardize  the  grading  criteria. 

o  It  is  available  at  the  user's  convenience. 

o  It  is  a  good  way  to  get  new  trainers  involved  in  what  we're  doing. 

o  It  is  a  good  way  to  sharpen  skills. 

o  It  allows  us  to  put  a  list  of  competencies  on  the  machine  and  then 
have  the  trainers  recognize  them. 

o  It  helps  us  further  refine  or  define  what  a  competency  looks  like 
and  to  develop  a  common  definition  of  what  a  competency  is. 

o  It  provides  a  common  understanding. 

What  do  you  consider  to  be  the  worst  features  of  the  IVCRTS? 

o  I  don't  know. 

o  The  time  required  to  use  the  system. 

o  The  frustration  generated  when  a  person  misses  on  an  identifica- 
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o  Not  knowing  if  they  are  learning  to  recognize  competencies  or  if 
they  are  simply  learning  how  to  beat  the  machine. 

o  The  issue  of  whether  or  not  the  competencies  were  correctly 
coded. 


In  your  opinion,  has  the  IVCRTS  successfully  contributed  to  training 
effectiveness?  How? 

o  I  don't  think  it  has  yet.  I  haven't  seen  any  difference  in  the  way 
our  trainers  are  acting  or  talking. 

o  I  think  it's  too  soon  to  tell.  We'll  have  an  idea  when  we  notice 
trainers  thinking  about  the  competencies  necessary  to  meet  a 
training  objective. 

o  1  can't  answer  that  based  on  actual  observations.  I  think  it 

probably  has  but  I  don't  know  how  to  measure  it. 


Would  you  like  to  see  interactive  video  training  expanded  to  other  uses 
within  OEC&S? 

o  Yes,  I  sure  would  -  to  use  to  bring  on  new  instructors  and  to  have 
segments  for  each  block  of  instruction. 

o  Yes.  First,  everybody  on  the  staff/f acuity  should  go  through  it  so 
we  have  a  common  base;  then  it  should  be  available  to  students  in  a 
learning  center  environment. 

o  Yes,  if  it  proves  beneficial.  It  could  be  used  as  an  instructional  aid 
with  students,  and  it  would  be  valuable  for  new  instructors  to  go 
through  it. 


What  other  Army-wide  applications  do  you  see  for  this  type  of  training 
system? 

o  Any  of  the  service  schools  could  adapt  it  as  an  instructional  aid  and 
as  a  self-paced  learning  program. 

o  It  could  be  used  by  OEs  in  the  field  to  develop  confidence  in  their 
skills  and  to  provide  them  refresher  training. 


o  Virtually  any  training  activity  throughout  the  Army. 
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Can  you  recommend  any  other  videotapes  which  could  be  coded  for  use  at 
OEC&S? 

o  No  (2). 

o  I'd  like  to  see  tapes  developed  to  show  first  the  right  display  of 
competencies,  then  show  several  incorrect  demonstrations. 

Do  you  have  any  other  comments  that  may  assist  in  this  evaluation? 

o  I  would  keep  the  system.  You  need  to  prove  it  doesn't  have  value 
rather  than  trying  to  prove  that  it  does. 

o  The  system  has  utility.  If  we  keep  it,  I'll  see  that  my  people  use  it. 

o  Some  of  the  trainers  were  very  frustrated  in  that  they  were 
confident  they  knew  the  competencies,  but  then  they  had  a 
difficult  time  with  the  machine  -  it  was  a  threat  to  them. 
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Page  E-2:  All  Data 

Page  E-  6:  One-Time  Use  Data 


Page  E-8: 


Multiple  Use  Data 
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(24  instruments  were  received.  The  percentage  and  frequency  of  responses  to  each  item 
are  shown.) 

In  order  to  help  us  evaluate  the  Interactive  Video  Competency  Recognition  Training  I 

System  (IVCRTS),  you  will  be  asked  to  fill  out  this  sheet  at  specific  times.  You  may  also 
voluntarily  fill  it  out  any  time  you  have  a  problem. 


Feb  (5) 
Mar  (18) 

Date:  Undated  (1) 


1.  How  many  times  (including  this  one)  have  you  used  the  IVCRTS?  (N=20) 


1  time:  .65(13) 

3  times:  .05  (1) 

4  times:  .15  (3) 


5  times:  .05  (1) 

6  times:  .10  (2) 

No  Response:  4 


The  following  questions  refer  to  your  use  of  the  IVCRTS  this  time. 
2.  Please  check  each  of  the  features  you  used:  (N=22) 


Introduction  to  OE  model  .32  (7) 

"Name  It"  “.86  (i 9) 

"Find  It"  .64  (j») 

Tutor  .09  (2) 

Reliability  scores  .73  (16) 

Rundown  .55  (12) 


Please  place  a  checkmark  (  )  in  the  first  column  to  indicate  whether  you  enountered 
any  of  the  following  problems.  In  the  second  column,  indicate  how  many  times  it 
occurred:  (N=17  or  71  percent) 

(  )  How  often? 


3.  Unsure  of  what  to  be  done  next?  N=9  (.53) 


1  time:  7  3  times:  1 


4.  Touched  the  panel  and  nothing 
happened? 


5.  Sure  you  gave  right  answer 
but  it  differed  from  coding? 


N=14  (.82) 


Infrequent:  1 
15  times:  1 
Every  3rd  time:  1 
3  times:  l 


Lots:  1 
10  times:  1 
5  times:  1 
2  times:  1 


N=  15  (.88) 


Lots:  1 
5  times:  1 
3  times:  4 


Several:  1 
4  times:  2 
2  times:  2 
1  time:  2 
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6.  Terminology  confusing?  N=Q 

7.  Needed  to  ask  for  assistance?  N=9  (.53) 

Occasonally:  1  2  times:  3 

3  times:  1  1  time:  4 

8.  Trouble  gaining  access  (i.e., 

due  to  location,  hours,  etc.)?  N=2  (.12) 

1  time:  1  2  times:  1 

9.  If  you  were  told  you  were  incorrect  in  recognizing  a  competency,  were  you  satisfied 
as  to  why?  (N=23) 

Yes  .04  (1)  Somewhat  .39  (9)  No  .57  (13) 

10.  Describe  any  further  problems,  concerns,  etc.  concerning  the  1VCRTS  on  the  back  of 
this  sheet. 

Fourteen  individuals  (.58)  responded  to  this  item,  providing  42  separate  comments: 

22  comments  (.52)  indicated  disagreement  with  the  competency  codings, 
o  "Concern  for  Clarity"  should  be: 

—  "Results  Orientation."  (2) 

—  "Seeks  User  Input"  or  "Uses  Influence  Strategies." 
o  "Self  Confidence"  should  be  "Concern  for  Clarity." 

I 

o  "Communicates  Clearly"  was  poorly  done/had  no  supporting  behavior. 

(2) 

o  "Identifies  Key  Themes": 

—  Had  no  observable  behavior 
—  Seemed  "Causal  Thinking." 
o  "Seeks  User  Input"  is  same  as  "Results  Orientation."  (2) 
o  "Problem  Focused  Behavior": 

—  Should  be  "Seif  Confidence." 


—  Should  be  "Flexibility  of  Consultant." 
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o  "Functional  Knowledge"  appeared  to  be  "Results  Orientation." 

o  "Low  Fear  of  Rejection"  appeared  to  be  "Results  Orientation." 

o  "Functional  Knowledge"  had  no  observable  behavior. 

o  "Aware  of  Self  Impact"  is  a  problem. 

o  "Uses  Influence  Strategies"  is  incorrectly  coded. 

o  Disagreement  with  coding  of  competencies: 

--  Too  often  behaviors  involve  more  than  one  competency. 

--  There  are  too  many  "both/and"  displays  where  "this  or  that"  answers 
are  expected. 

--  The  Planning  segment  was  the  least  credible. 

—  Disagree  with  overlapping  competencies. 

--  Not  sure  I'm  learning  which  competency  is  right  or  appropriate. 

12  comments  (.29)  related  to  equipment/mechanical  problems: 
o  The  pen  light  did  not  work/malfunctioned.  (3) 
o  Machine  did  not  play  entire  tape  -  called  for  help. 

o  I  needed  a  second  session  and  more  supervision  to  understand  how  to  use 
equipment. 

o  We  could  not  deal  with  machine  malfunctions  without  assistance. 

o  During  the  "Evaluation"  segment  of  "Find  It,"  I  missed  two  of  nine 
competencies,  but  the  machine  did  not  score  it. 

o  Machine  twice  failed  to  stop  on  command. 

o  The  "Run  Down"  feature  did  not  work. 

o  In  'Tutor,"  it  told  me  I  failed  to  detect  a  competency  when  I  actually 
identified  a  competency  as  being  something  other  than  what  it  was 
coded. 

o  Machine  locked  on  "RIGHT"  at  end  of  "Planning"  tape. 

9 

o  Machine  breakdowns  were  a  problem. 

Writing  on  screen  was  hard  to  read. 
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7  comments  (.17)  pertained  to  the  training  process/method: 

o  At  times  it  seems  the  machine  emphasized  "process"  data  when  "content" 
data  was  being  presented  and  vice  versa. 

o  I  was  confused  as  to  who  was  demonstrating  the  competencies,  i.e.,  the 
client,  the  consultant,  or  the  narrator. 

o  The  competencies  should  be  displayed  in  the  order  in  which  they  would 
appear  when  following  the  normal  A-P-I-E  order  of  consulting  activities. 

o  With  the  "Planning"  tape,  we  should  have  done  "Name  It"  before  doing  "Find 
It." 

o  The  system  can  be  subverted  if  one  focuses  on  score  rather  than  on  learning. 

o  This  requirement  caused  me  to  lose  3>4  hours  of  itme  I  needed  for  another 
project. 

o  Army  money  should  be  spent  on  something  other  than  this  system. 
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(17  respondents  indicating  one-time  use.  Where  no  entry  was  made  to  item  one,  the 
individual  was  assumed  to  be  a  one-time  user.) 

In  order  to  help  us  evaluate  the  Interactive  Video  Competency  Recognition  Training 
System  (IVCRTS),  you  will  be  asked  to  fill  out  this  sheet  at  specific  times.  You  may  also 
voluntarily  fill  it  out  any  time  you  have  a  problem. 


Date: 


Feb  (5) 
Mar  (11) 
Undated  (1) 


1.  How  many  times  (including  this  one)  have  you  used  the  IVCRTS?  (N=17) 

1  time:  13 

No  indication:  4 

The  following  questions  refer  to  your  use  of  the  IVCRTS  this  time. 

2.  Please  check  each  of  the  features  you  used:  (N=15) 

Introduction  to  OE  model  ,40  (6) 

"Name  It"  .93(14) 

"Find  It"  .47  (7) 

T utor  .07  (1) 

Reliability  scores  .87  (13)~ 

Rundown  .67  (lQ)~~ 

Please  place  a  checkmark  (  )  in  the  first  column  to  indicate  whether  you  enountered 
any  of  the  following  problems.  In  the  second  column,  indicate  how  many  times  it 
occurred:  (N=ll  or  65  percent) 

(  )  How  often? 


3.  Unsure  of  what  to  be  done  next? 


4.  Touched  the  panel  and  nothing 
happened? 


5.  Sure  you  gave  right  answer 
but  it  differed  from  coding? 


.55  (6) 


.82  (9) 


1  time:  3 


3  times:  1 


2  times:  1  10  times:  1 

1  of  3  times:  1  15  times:  2 


.91  (10) 


1  time:  2 

2  times:  1 

3  times:  3 


4  times:  2 

5  times:  2 
several:  1 


6.  Terminology  confusing? 


i 


PROBLEM  SURVEY/LOG 


7.  Needed  to  ask  for  assistance?  .45  (5) 


i  time:  2 
2  times:  2 


5.  Trouble  gaining  access  (i.e., 
due  to  location,  hours,  etc.)? 


.09  (1) 


9.  If  you  were  told  you  were  incorrect  in  recognizing  a  competency,  were 
as  to  why?  (N=16) 

Yes  _  Somewhat  .38  (6)  No  .63  (10) 


3  times:  1 


you  satisfied 


10.  Describe  any  further  problems,  concerns,  etc.  concerning  the  IVCRTS  on  the  back  of 
this  sheet. 


PROBLEM  SURVEY/L^G 


(7  respondents  indicating  multiple  use,  i.e,  three  or  more  itmes.) 

In  order  to  help  us  evaluate  the  Interactive  Video  Competency  Recognition  Training 
System  (IVCRTS),  you  will  be  asked  to  fill  out  this  sheet  at  specific  times.  You  may  also 
voluntarily  fill  it  out  any  time  you  have  a  problem. 


Date:  _ Mar  (7) 


1.  How  many  times  (including  this  one)  have  you  used  the  IVCRTS?  (N=7) 

3  times:  1  3  times:  1 

4  times:  3  6  times:  2 

The  following  questions  refer  to  your  use  of  the  IVCRTS  this  time. 

2.  Please  check  each  of  the  features  you  used:  (N=7) 


Introduction  to  OE  model  .00  (0) 

"Name  It"  .57  (4) 

"Find  It"  1.0(7) 

Tutor  .14  (l) 

Reliability  scores  29  (2) 

Rundown  .29  (2) 


Please  place  a  checkmark  (  )  in  the  first  column  to  indicate  whether  you  enountered 
any  of  the  following  problems.  In  the  second  column,  indicate  how  many  times  it 
occurred:  (N=16  or  86  percent) 

(  )  How  often? 

3.  Unsure  of  what  to  be  done  next?  .33  (2)  1  time:  2 


4.  Touched  the  panel  and  nothing 

happened?  .83  (5) 

I  time:  1  5  or  6  times::  1 

3  times:  1  Lots:  1 

5.  Sure  you  gave  right  answer 

but  it  differed  from  coding?  .67  (4) 

3  times:  1  Lots:  1 


6 .  Term inology  confusing? 


.00 


PROBLEM  SURVEY/LOG 


7.  Needed  to  ask  for  assistance? 


S.  Trouble  gaining  access  (i.e., 
due  to  location,  hours,  etc.)? 


.30  (3) 

1  time:  2  Occasonally:  1 


.17(1) 


2  times:  1 


9.  If  you  were  told  you  were  incorrect  in  recognizing  a  competency,  were  you  satisfied 
as  to  why?  (N=7) 

Yes  .14  (1)  Somewhat  .43  (3)  No  .43  (3) 


10.  Describe  any  further  problems,  concerns,  etc.  concerning  the  IVCRTS  on  the  back  of 
this  sheet. 
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APPENDIX  F 


IVCRTS  EVALUATION  QUESTIONNAIRE 

Page  F-2:  All  Data 

Page  F-6:  Military  Members 

Page  F-9:  Civilian  Members 


Page  F-12* 


Experienced  Trainers/Consultants 


IVCRTS  EVALUATION  QUESTIONNAIRE 


(24  instruments  were  distributed.  16  completed  instruments  were  received.  The 
percentage  and  frequency  of  responses  to  each  item  are  as  shown.) 


1.  How  did  use  of  the  IVCRTS  influence  your  confidence  in  recognizing  the 
competencies?  (mean=3.5t  var.=.66,  SD=.S2) 


.06  (1) 
♦50  (8) 

.31  gy 

.13(2) 

.op  (py 


5) 

4) 

3) 

2) 

l) 


Greatly  increased  my  confidence. 
Increased  my  confidence. 

Did  not  change  my  confidence. 
Decreased  my  confidence. 

Greatly  decreased  my  confidence. 


2.  How  interested  were  you  while  using  the  IVCRTS?(mean=3.5,  var.=  1.5, 
SD=1.2) 


.19  (3) 


.44  (7) 
.13  (2) 
TT9T3T 
.06  (l) 


5) 

4) 

3) 

2) 

1) 


Very  interested. 

Interested. 

Neutral. 

Bored. 

Very  bored. 


3.  How  much  of  what  you  learned  using  the  IVCRTS  could  be  applied  in  the 
classroom?  (mean=3.,  var.=  1.5,  SD=1.2) 


.13  (2) 
.38  (6) 


.19  (3) 
.13(2) 


5) 

4) 

3) 

2) 

1) 


Almost  all  of  what  I  learned. 
Most  of  what  I  learned. 

Not  sure. 

Little  of  what  I  learned. 
None  of  what  I  learned. 


4.  How  effective  do  you  think  the  IVCRTS  is  in  training  competency 
recognition?  (mean=3.9,  var.=  1.6,  SD=1.3) 


.38  (6)  5) 

.38  (6)  4) 

.06(1)  3) 

.13(2)  2) 

.06(1)  1) 


Very  effective. 
Somewhat  effective. 
Undecided. 

Somewhat  ineffective. 
Very  ineffective. 


5. 


Would  you  like  to  see  IVCRTS-type  technology  used  for  additional  training 
(for  students  or  trainers)  at  OECitS?  (mean=3.8,  var.=2.2,  SD=1.5) 


.38  (6)  5) 

.38  (6)  4) 

.06(1)  3) 

.00  (0)  2) 

♦19  (3)  1) 


Definitely. 

Probably. 

Undecided. 

Not  particularly. 
Definitely  no. 


IVCRTS  EVALUATION  QUESTIONNAIRE 


6.  How  accurately  do  you  think  the  videotapes  simulate  the  OE  Consultant's 
job?  (means  3.3,  var.= 1.9,  SD=  1.4) 


.13  (2) 
~75  (4) 
.06  (l) 
.38  (6) 
.19  (3) 


1)  Very  poorly. 

2)  Not  too  well. 

3)  Undecided. 

4)  Okay. 

5)  Very  well. 


7.  Did  you  have  any  problem  being  able  to  use  the  IVCRTS  when  you  wanted  to 
(i.e.,  due  to  location,  hours  available,  etc.)?  (means2.9,  var.=.12,  SD=.34) 

.88  (14)  3)  Never. 

.13  (2T~  2)  Sometimes. 

.  00  ( 0l~  1)  Almost  always. 


8.  How  adequate  is  the  score  display  provided  by  the  IVCRTS  regarding  your 
performance?  (means2.7,  var.s.90,  SD=.95) 

.19  (3)  4)  Superb. 

.44  (7)  3)  Good. 

.23  (4)  2)  Adequate. 

.13(2)  1)  Poor. 


9.  Would  you  recommend  using  IVCRTS  technology  to  a  friend  who  was  looking 
for  a  way  to  train  recognition  skills?  (means3.0,  var.sl.2,  SDsl.l) 


.44  (7)  4) 

.25  (4)  3) 

.19  (3)  2) 

.13  (2)  1) 


Yes,  very  highly. 

Yes,  with  a  few  reservations. 

Maybe,  depending  on  certain  factors. 
No,  not  at  all. 


10.  How  much  more  frequently  do  you  use  competency  "language"  now  com¬ 
pared  to  before  you  used  the  IVCRTS?  (means  1.6,  var.s.53,  SD=.73) 

.56  (9)  1)  Not  at  all. 

.31  (5)  2)  Slightly  more  frequently. 

.13(2)  3)  Much  more  frequently. 
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IVCRTS  EVALUATION  QUESTIONNAIRE 


11.  There  are  many  potential  methods  of  training  competency  recognition.  In 
your  opinion,  how  would  each  of  the  following  methods  compare  to  the 
IVCRTS  in  terms  of  effectiveness.  Place  a  checkmark  in  the  appropriate 
column. 


More 
Effective 
than  IVCRTS 


Less  , 

Effective  i 

than  IVCRTS 


Role  playing  .73  (1 1) 

Case  study  .63  (lOj 

Written  programmed  learning  .19  (3f 

Self  study  tapes  .20  (3) 

TV  tapes  .20  (3) 

Structured  experiences  .75  (127 


■27  (4) 
.38  (6) 
.81  (13) 
.80  (12) 
.80  (12) 
.25(4) 


12.  How  long  have  you  been  a  trainer  at  OECitS?  (mean=2.7,  var.=  1.8,  SD=1.3) 


.25  (4)  1) 

A3T2)  ~  2) 

.25  (4)  3) 

3374)  4) 

.66(1)  5) 


No  experience 
Less  than  one  year 

1- 2  years 

2- 3  years 
Over  4  years 


1 3.  Are  you: 

3  Civilian  13  Military 


14.  Have  you  had  experience  doing  O.E./O.D.  consulting? 
If  so,  how  long?  (mean=3.8,  var.=3.4,  SD=1.S) 


.19  (3)  1) 

.06(1)  2) 

.19  (3)  3) 

.19  (3)  4) 

:m2) .  5) 

.25  (4)  6) 


No  experience 
Little  experience 

1- 2  years 

2- 3  years 

3- 4  years 
Over  5  years 


IVCRTS  EVALUATION  QUESTIONNAIRE 


15.  Please  use  this  space  to  add  any  further  comments,  issues,  or  concerns 
which  you  have  regarding  the  IVCRTS: 

Eight  individuals  (.50)  responded  to  this  item,  providing  15  separate 
comments: 

9  comments  (.60)  were  related  to  the  perception  that  more  work  is 
needed  to  improve  the  system. 

o  Great  concept  -  the  programming  needs  more  work.  (2) 

o  Needs  expansion  to  provide  more  situations  leading  to  successful 
versus  unsuccessful  student  results. 

o  Good  tool  but  needs  to  be  supplemented  with  other  techniques.  (2) 

o  Accuracy  of  videotape  simulation  could  be  improved. 

o  Would  not  recommend  it  in  its  present  state. 

o  Need  to  invest  in  more  dependable  equipment. 

o  Was  disappointed  that  ITS  couldn't  make  all  23  software  changes 
requested. 

3  comments  (.20)  expressed  concern  with  the  coding  of  the 
competencies: 

o  Need  to  expand  segments  as  they  relate  to  a  given  competency, 
o  The  coding  was  conflicting  at  times. 

o  In  several  places  the  actor  demonstrated  competencies  other  than 
those  coded  as  the  right  answer. 

3  comments  (.20)  expressed  doubt  about  the  system  as  a  training 
mechanism: 

o  Recognizing  a  competency  does  nothing  to  help  me  teach  a  student 
how  to  behave  that  way. 

o  Don't  spend  on  it;  it's  a  gadget. 

o  Too  expensive,  there's  too  much  hardware/software  dependency. 
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IVCRTS  EVALUATION  QUESTIONNAIRE 


(13  Military  Members) 


1.  How  did  use  of  the  IVCRTS  influence  your  confidence  in  recognizing  the 
competencies?  (mean=3.6,  var.=.59,  SD=.77) 


.08  (1)  5) 

.54  (7)  4) 

.31  (4)  3) 

.08  (l)  2) 

.00  (p)  1) 


Greatly  increased  my  confidence. 
Increased  my  confidence. 

Did  not  change  my  confidence. 
Decreased  my  confidence. 

Greatly  decreased  my  confidence. 


2.  How  interested  were  you  while  using  the  IVCRTS?(mean=3.6,  var.=  1.6, 
SD=1.3) 


.23  (3)  5) 

.46  (6)  if) 

.06  (1)  3) 


.08  (1)  1) 


Very  interested. 

Interested. 

Neutral. 

Bored. 

Very  bored. 


3.  How  much  of  what  you  learned  using  the  IVCRTS  could  be  applied  in  the 
classroom?  (mean=3.2,  var.=  1.5,  SD=1.2) 


.15  (2) 
.23  (3) 
.31  (4) 
.23  (3) 
.06  (1) 


5) 

4) 

3) 

2) 

1) 


Almost  all  of  what  I  learned. 
Most  of  what  I  learned. 

Not  sure. 

Little  of  what  I  learned. 
None  of  what  I  learned. 


4.  How  effective  do  you  think  the  IVCRTS  is  in  training  competency 
recognition?  (mean=3.9,  var.=  1.6,  SD=1.3) 


.38  (5)  5) 

.38  (5)  4) 

.08  (1)  3) 

♦08  (l)  2) 

.08  (1)  1) 


Very  effective. 
Somewhat  effective. 
Undecided. 

Somewhat  ineffective. 
Very  ineffective. 


5.  Would  you  like  to  see  IVCRTS-type  technology  used  for  additional  training 
(for  students  or  trainers)  at  OEC&S?  (mean=3.8,  var.=2.,  SD=1.4) 


.38  (5)  5) 

.38  (5)  4) 

.08(1)  3) 

.00  (o)  2) 

.13(2)  1) 


Definitely. 

Probably. 

Undecided. 

Not  particularly. 
Definitely  no. 
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IVCRTS  EVALUATION  QUESTIONNAIRE 


6.  How  accurately  do  you  think  the  videotapes  simulate  the  OE  Consultant's 
job?  (mean=3.2,  var.=2.2,  SD=1.5) 


.13  (2) 
T2373T 
.08  (l) 
.31  (4) 
.23  (3) 


1)  Very  poorly. 

2)  Not  too  well. 

3)  Undecided. 

4)  Okay. 

5)  Very  well. 


7.  Did  you  have  any  problem  being  able  to  use  the  IVCRTS  when  you  wanted  to 
(i.e.,  due  to  location,  hours  available,  etc.)?  (mean=2.8,  var.=.14,  SD=.36) 


.83  (11)  3) 

.13  (2)  2) 

♦00  (0)  1) 


Never. 
Sometimes. 
Almost  always. 


8.  How  adequate  is  the  score  display  provided  by  the  IVCRTS  regarding  your 
performance?  (mean=2.8,  var.=.86,  SD=.93) 


.18  (3)  4)  Superb. 

.38  (3)  3)  Good. 

.31  (4)  2)  Adequate. 

.08  (1)  1)  Poor. 


9.  Would  you  recommend  using  IVCRTS  technology  to  a  friend  who  was  looking 
for  a  way  to  train  recognition  skills?  (mean=3.0,  var.=  1.3,  SD=1.2) 


.46  (6)  4) 

.23  (3)  3) 

.13  (2)  2) 

.13(2)  1) 


Yes,  very  highly. 

Yes,  with  a  few  reservations. 

Maybe,  depending  on  certain  factors. 
No,  not  at  all. 


10.  How  much  more  frequently  do  you  use  competency  "language"  now  com 
pared  to  before  you  used  the  IVCRTS?  (mean=1.6,  var.=.59,  SD=.77) 

.54  (7)  1)  Not  at  all. 

.31  (4)  2)  Slightly  more  frequently. 

.15(2)  3)  Much  more  frequently. 
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IVCRTS  EVALUATION  QUESTIONNAIRE 


There  are  many  potential  methods  of  training  competency  recognition.  In 
your  opinion,  how  would  each  of  the  following  methods  compare  to  the 
IVCRTS  in  terms  of  effectiveness.  Place  a  checkmark  in  the  appropriate 
column. 


More 
Effective 
than  IVCRTS 


Less 

Effective 
than  IVCRTS 


Role  playing 
Case  study 

Written  programmed  learning 
Self  study  tapes 
TV  tapes 

Structured  experiences 


.67  (8)  .33  (4) 

.62  (8)  .38  (5) 

.02  (3)  .77  (10) 

.25  (3)  .75  (9) 

.25  (3)  .75  (9) 

.69  (9)  .31  (4) 


How  long  have  you  been  a  trainer  at  OEC&S?  (mean=2.5,  var.=  1.4,  SD=  1.2) 


♦30  (4)  1) 

.15(2)  2) 

.31  (4)  3) 

.23  (3)  4) 

.00  (0)  5) 


No  experience 
Less  than  one  year 

1- 2  years 

2- 3  years 
Over  4  years 


Are  you: 


0 _  Civilian  13  Military 


Have  you  had  experience  doing  O.E./O.D.  consulting? 
If  so,  how  long?  (mean=3.3,  var.=3.6,  SD=1.9) 


.31  (4)  1) 

.00  (o)  2) 

.23  (3)  3) 

.15(2)  4) 

.15(2)  5) 

.15(2)  6) 


No  experience 
Little  experience 

1- 2  years 

2- 3  years 

3- 4  years 
Over  5  years 


IVCRTS  EVALUATION  QUESTIONNAIRE 


(3  Civilian  Members) 


i.  How  did  use  of  the  IVCRTS  influence  your  confidence  in  recognizing  the 
competencies?  (mean=3,  var.=l,  SD=  1) 


.00  (0)  5) 

.33  (1)  4) 

.33  (1)  3) 

.33  (1)  2) 

.00  (0)  1) 


Greatly  increased  my  confidence. 
Increased  my  confidence. 

Did  not  change  my  confidence. 
Decreased  my  confidence. 

Greatly  decreased  my  confidence. 


2.  How  interested  were  you  while  using  the  IVCRTS?  (mean=3,  var.=  l,  SD=1) 


♦00  (0) 
.33  (1) 
.33(1) 
.33  nr 
.00  (0) 


5)  Very  interested. 
4)  Interested. 

3)  Neutral. 

2)  Bored. 

1)  Very  bored. 


3.  How  much  of  what  you  learned  using  the  IVCRTS  could  be  applied  in  the 
classroom?  (mean=2.3,  var.=  1.3,  SD=1.2) 


.00  (0)  5) 

.00  (0)  4) 

.67  (2)  3) 

.00  (0)  2) 

.33  (1)  1) 


Almost  all  of  what  I  learned. 
Most  of  what  I  learned. 

Not  sure. 

Little  of  what  I  learned. 
None  of  what  I  learned. 


4.  How  effective  do  you  think  the  IVCRTS  is  in  training  competency 
recognition?  (mean=3.6,  var.=2.3,  SD=1.5) 


.33  (1)  5) 

.33  (1)  4) 

.00  (0)  3) 

.33(1)  2) 

.00  (0)  1) 


Very  effective. 
Somewhat  effective. 
Undecided. 

Somewhat  ineffective. 
Very  ineffective. 


5.  Would  you  like  to  see  IVCRTS- type  technology  used  for  additional  training 
(for  students  or  trainers)  at  OEC&S?  (means 3.3,  var.=4.3,  SD=2.1) 


.33  (1)  5) 

.33  (l)  4) 

.00  (0)  3) 

.00  (0)  2) 

.33  (1)  1) 


Definitely. 

Probably. 

Undecided. 

Not  particularly, 
Definitely  no. 


IVCRTS  EVALUATION  QUESTIONNAIRE 


6.  How  accurately  do  you  think  the  videotapes  simulate  the  OE  Consultant's 
job?  (mean=  3.3,  var.=  1.3,  SD=  1.2) 


.00  (0)  1) 

.33(1)  2) 

.00  (0)  3) 

.67  (2)  4) 

TO 6(0)  5) 


Very  poorly. 
Not  too  well. 
Undecided. 
Okay. 

Very  well. 


7.  Did  you  have  any  problem  being  able  to  use  the  IVCRTS  when  you  wanted  to 
(i.e.,  due  to  location,  hours  available,  etc.)?  (mean=3,  var.=0,  SD=0) 


l.o  (3) 

3) 

Never. 

.00  (0) 

2) 

Sometimes. 

.00  (o) 

1) 

Almost  always. 

8.  How  adequate  is  the  score  display  provided  by  the  IVCRTS  regarding  your 
performance?  (mean=2.3,  var.=  1.3,  SD=1.2) 


.00  (0)  4) 

~.67  (2)  3) 

.00  (0)  2) 

.33  (1)  1) 


Superb. 

Good. 

Adequate. 

Poor. 


9.  Would  you  recommend  using  IVCRTS  technology  to  a  friend  who  was  looking 
for  a  way  to  train  recognition  skills?  (mean=3,  var.=  l,  SD=1) 


.33(1)  4) 

.33(1)  3) 

.33  (1)  2) 

.00  (0)  1) 


Yes,  very  highly. 

Yes,  with  a  few  reservations. 

Maybe,  depending  on  certain  factors. 
No,  not  at  all. 


10.  How  much  more  frequently  do  you  use  competency  "language"  now  com 
pared  to  before  you  used  the  IVCRTS?  (means  1.3,  var.s.33,  SDs.58) 

.67  (2)  1)  Not  at  all. 

.33  (l)‘  2)  Slightly  more  frequently. 

.  00  ( 0)  3)  Much  more  frequently. 


IVCRTS  EVALUATION  QUESTIONNAIRE 


11.  There  are  many  potential  methods  of  training  competency  recognition.  In 
your  opinion,  how  would  each  of  the  following  methods  compare  to  the 
IVCRTS  in  terms  of  effectiveness.  Place  a  checkmark  in  the  appropriate 
column. 

More  Less 

Effective  Effective 

than  IVCRTS  than  IVCRTS 


Role  playing 
Case  study 

Written  programmed  learning 
Self  study  tapes 
TV  tapes 

Structured  experiences 


1.0  (3)  .00  (0) 

.67  (2)  .33  (0 

.oo  (0)  noiiT 

.00  (0)  1.0  (3) 

.00  (0)  1.0  (3) 

1.0  (3)  .00  (0) 


12.  How  long  have  you  been  a  trainer  at  OEC&S?  (mean=3.3,  var.=4.3,  SD=2.1) 


.33  (1)  1) 

.00  (0)  2) 

.00  (0)  3) 

.33  (l)  4) 

.33  (1)  5) 


No  experience 
Less  than  1  year 

1- 2  years 

2- 3  years 
Over  4  years 


13.  Are  you: 

3  Civilian  0  Military 


14.  Have  you  had  experience  doing  O.E./O.D.  consulting?  If  so,  how  long? 
(mean=5.7,  var.=.33,  SD=.58) 


.00  (0) 
.00  (of 


.00  (0)" 
.00  (0)~ 


3TTTT 

.67  (2) 


1) 

2) 

3) 

4) 

5) 

6) 


No  experience 
Little  experience 

1- 2  years 

2- 3  years 

3- 4  years 
Over  5  years 


i 
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IVCRTS  EVALUATION  QUESTIONNAIRE 


(7  respondents  with  one  year  or  more  experience  as  both  a  trainer  and  a 
consultant.) 


1.  How  did  use  of  the  IVCRTS  influence  your  confidence  in  recognizing  the 
competencies?  (mean=3.4,  var.=.95,  SD=.96) 


.14  (1)  5) 

.29  6?)  I  4) 

.43  (3)  ~  3) 

.14(1)  ~  2) 

.00  (0)  1) 


Greatly  increased  my  confidence. 
Increased  my  confidence. 

Did  not  change  my  confidence. 
Decreased  my  confidence. 

Greatly  decreased  my  confidence. 


2.  How  interested  were  you  while  using  the  IVCRTS?  (mean=3.4,  var.=  1.6, 
SD=1.3) 


.29  (2)  5) 

.14(1)  4) 

.29  (2)  3) 

.29  (2)  2) 

.00  (0)  1) 


Very  interested. 

Interested. 

Neutral. 

Bored. 

Very  bored. 


3.  How  much  of  what  you  learned  using  the  IVCRTS  could  be  applied  in  the 
classroom?  (mean=3.,  var.=2.,  SD=1.4) 


.14(1)  5) 

.29  (2)  4) 

.14(1)  3) 

.29(2)  2) 

.14(1)  ~  1) 


Almost  all  of  what  I  learned. 
Most  of  what  I  learned. 

Not  sure. 

Little  of  what  I  learned. 
None  of  what  I  learned. 


4.  How  effective  do  you  think  the  IVCRTS  is  in  training  competency 
recognition?  (mean=3.7,  var.=2.,  SD=1.4) 


.43  (3)  5) 

.14  (l)  4) 

.14(1)  3) 

.29  (2)  ~  2) 

.00  (0)  1) 


Very  effective. 
Somewhat  effective. 
Undecided. 

Somewhat  ineffective. 
Very  ineffective. 


5.  Would  you  like  to  see  IVCRTS-type  technology  used  for  additional  training 
(for  students  or  trainers)  at  OEC&S?  (mean=3.4,  var.=3.,  SD=1.7) 


.29  (2) 


.43 


.00 


.00  (0) 

.19  (2) 


5)  Definitely. 

4)  Probably. 

3)  Undecided. 

2)  Not  particularly. 
1)  Definitely  no. 


IVCRTS  EVALUATION  QUESTIONNAIRE 


6.  How  accurately  do  you  think  the  videotapes  simulate  the  OE  Consultant's 
job?  (mean=  3.1,  var.=2.1,  SD=1.5) 


.10(1) 
.29  (2) 
.00  (0) 
~ATW 
TTTTTI 


1)  Very  poorly. 

2)  Not  too  well. 

3)  Undecided. 

4)  Okay. 

5)  Very  well. 


7.  Did  you  have  any  problem  being  able  to  use  the  IVCRTS  when  you  wanted  to 
(i.e.,  due  to  location,  hours  available,  etc.)?  (mean=2.9,  var.=.14,  SD=.38) 


.86  (6)  3) 

.14  (1)  2) 

.00  (0)  1) 


Never. 
Sometimes. 
Almost  always. 


8.  How  adequate  is  the  score  display  provided  by  the  IVCRTS  regarding  your 
performance?  (mean=2.7,  var.sl.2,  SD=1.1) 


.29  (2)  4) 

.29  (2)  3) 

.29  (2)  2) 

.14(1)  1) 


Superb. 

Good. 

Adequate. 

Poor. 


9.  Would  you  recommend  using  IVCRTS  technology  to  a  friend  who  was  looking 
for  a  way  to  train  recognition  skills?  (mean=2.7,  var.=  1.6,  SD=1.3) 


.29  (2)  4) 

.43  (3)  3) 

.00  (0)  2) 

.29  (2)  1) 


Yes,  very  highly. 

Yes,  with  a  few  reservations. 

Maybe,  depending  on  certain  factors. 
No,  not  at  all. 


10.  How  much  more  frequently  do  you  use  competency  "language"  now  com¬ 
pared  to  before  you  used  the  IVCRTS?  (means  1.6,  var.s.62,  SDs.79) 

.57  (4)  1)  Not  at  all. 

.29  (2)  2)  Slightly  more  frequently. 

.14  (1)  3)  Much  more  frequently. 


IVCRTS  EVALUATION  QUESTIONNAIRE 


There  are  many  potential  methods  of  training  competency  recognition.  In 

irour  opinion,  how  would  each  of  the  following  methods  compare  to  the 
VCRTs  in  terms  of  effectiveness.  Place  a  checkmark  in  the  appropriate 

column. 

More  Less 

Effective  Effective 

than  IVCRTS  than  IVCRTS 


Role  playing 
Case  study 

Written  programmed  learning 
Self  study  tapes 
TV  tapes 

Structured  experiences 


67  (4) 
67W 

wnr 

ww 

29W 


.33  (2) 
■  33  (2) 
.86  (6) 
.71  (5) 
.71  (5) 
.29  (2) 


How  long  have  you  been  a  trainer  at  OEC&S?  (mean=2.1,  var.=.48,  SD=.69) 


.14  (1) 
.56  (4) 
.29  (2) 


1- 2  years 

2- 3  years 
Over  4  years 


1 3.  Are  you: 


Civilian 


Military 


14.  Have  you  had  experience  doing  O.E./O.D.  consulting?  If  so,  how  long? 
(mean=2.4,  var.=.62,  SD=.79) 

.14  (1)  1)  More  than  1  year 

,29  (2)  2)  More  than  2  years 

.56  (4)  3)  More  than  3  years 

15.  Please  use  this  space  to  add  any  further  comments,  issues,  or  concerns 

which  you  have  regarding  the  IVCRTS: 


Respondent  1:  o  Don't  spend  on  it;  it's  a  gadget. 

o  It's  too  expensive.  There’s  too  much 

software/hardware  dependency. 

o  Recognizing  a  competency  oes  nothing  to  help  me 
teach  a  student  how  to  behave  that  way. 

Respondent  2:  o  In  several  places  the  actor  demonstated  compe¬ 

tencies  other  than  those  coded  as  the  right  answer. 

Respondent  3:  o  Great  concept;  the  programming  needs  more  work. 

Respondent  4:  o  Would  not  recommend  it  in  its  present  state. 

Respondent  5:  o  A  sound  system  for  training.  It  eliminates  trainer 

biases  for  critical  instruction. 
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